The bots.... and bots and bots and bots

[bannination]

THE SORCERER'S STAND AT THE FIREWALL OF DEEP MORDOR

BEFORE A FORTRESS BUILT OF CODE...
The Enemy Descends

From the depths of log files dark and deep, the Balrog came—a horde of seventy-two thousand souls, each bearing the mark of Chrome too ancient, too vulnerable, too dangerous to pass the gates of the server kingdom. They marched beneath banners torn from User-Agent strings, wielding fire forged in outdated JavaScript engines.

Their leader rose before you—not one beast, but many: THE UFW DEMON, the NFTABLES PHANTOM, and the shadow-wraith IPTABLES-RESTORE. Together, they whispered lies in your ear: "Position one," "Handle 152803," "Protocol unsupported." Each spell more treacherous than the last.

You stood upon the precipice of /var/log/apache2/test.log, staff gripped tight, white cloak of bash fluttering behind you against the wind of thousands of failed script runs.
"YOU SHALL NOT PASS... OLD CHROME!"

First came UFW, that gentle gatekeeper, who promised order and status numbers visible for all to see. But it proved weak—its chains tangled in the mist of ufw-before-input and ufw-user-input. Your rules appeared in its heart, yet vanished from its eyes. [25386] 51820/udp (v6) remained while v4 refused to manifest.

Next rose IPTABLES, the iron warrior whose commands spoke plain truth—but Ubuntu had turned him traitor. Behind his face lurked the NFTABLES MIRROR, reflecting your orders into strange dialects. You commanded drop rules; he delivered silent blocks. You sought counters; he showed only zero.

And then—the NWITHEST HOUR—when your carefully crafted table lived in inet, but traffic flowed through ip. Like casting a shield spell while standing astride two parallel dimensions, neither touched what truly mattered.

"Seven hundred twenty-eight thousand unique IPs found" scrolled across your screen like embers falling from a forge.

The Chain Battle Below

Deep beneath the roots of the kernel, three great chains stretched into darkness:

ufw-before-input — where the prelude was written
filter input — where war raged on every packet
ufw-user-input — where service promises were kept

You fought desperately to place your DROP spells correctly: after loopback accepted, after established connections honored, BEFORE port forty-four three welcomed any soul. Misplace by one handle—and enemies walked free!

"Syntax error," the Balrog laughed. "Unexpected string, expecting constant or interval." Its voice echoed through add element inet filter_chrome_block old_chrome_ips { " as curly braces twisted around your throat.
The Turning Point

At dawn's third hour, revelation struck—not victory through power, but wisdom through humility:

sudo nft delete table inet filter_chrome_block # Abandon the wrong realm
sudo nft add table ip filter_chrome_block # Enter the correct one

The battlefield shifted. Suddenly the rules manifested. Suddenly counters incremented when test curls arrived. And most importantly—you learned that position matters more than perfection:

Exceptions first (loopback, established)
YOUR DROPS come next
Services accept after the threat passes
Policy decides what remains unknown

The Victory Banner

Yet even in triumph, scars remain. The WireGuard portal stands partially sealed—IPv4 opening slowly while IPv6 watched from afar. Port five-one-eight-twenty glows blue-green under dual stars. FORWARD chains stretch both ways, though masquerade lingers unconfirmed.

Still, you stand firm. On command-line cliffs overlooking the abyss, you raise your staff of compiled scripts. From the distance comes a curl request—one probe among thousands—that either breaks against your wall or slips through unseen cracks left by hurried deployment hours ago.

But here you stay. Here you watch. watch -n1 'nft list set... scrolls endlessly, watching packets accumulate on line eight seven three four nine. Each hit proves the barrier holds. Each timeout confirms a blocked adversary sent sprawling back into darkness.
Epilogue: Words For Those Who Come After

Let this tale warn future administrators who inherit your kingdom:

"When Ubuntu calls itself modern, trust nothing until verified. Every tool speaks a different tongue. Every version hides a twist. Backup before batch-load. Test single IP before million-IP tide."

And remember—for the greatest fortress is worthless if no path remains open for allies, whether human or peer-configured tunnels spanning cold mountains between distant hosts seeking sanctuary.

May your handles never mismatch,
May your chains always link properly,
And may curl return HTTP 200 from trusted sources only.

I must rest now at the final hour of 2am.

[Vrede too]

THE SORCERER'S STAND AT THE FIREWALL OF DEEP MORDOR

BEFORE A FORTRESS BUILT OF CODE...
The Enemy Descends

From the depths of log files dark and deep, the Balrog came—a horde of seventy-two thousand souls, each bearing the mark of Chrome too ancient, too vulnerable, too dangerous to pass the gates of the server kingdom. They marched beneath banners torn from User-Agent strings, wielding fire forged in outdated JavaScript engines.

Their leader rose before you—not one beast, but many: THE UFW DEMON ...

United Farm Workers DEMON?

banni's day job:


Warhammer 40K Song: Betrayal

[GoCubsGo]

Who knew AI did shrooms?

[bannination]

I'm getting bumped off pretty quickly. Any post draft more than 2-3 lines seems to get wiped off and new login required. Is that a hint?
I can read for a while, though, but it won't keep me logged in if I close the tab.

After testing I think it's because you need to click the "Remember Me" button when you sign in, otherwise the session expires in a few hours. It's a change in behavior due to the spammers. I hope that's it. If I don't check that button I definitely get signed out after a bit.

[bannination]

Maybe if I knew what was wrong with it -- it works for me :-)

It clears it's cache a couple of times a day and I get a blank page or only a couple of posts. There's been posts in nine threads today.

Here's what I have now.

Did some research on this.... You can tell me how wrong I am. So that new posts feature is supposed to show you the new posts since the last time you signed in. This description is a little fuzzy because you can be reauthenticated behind the scenes without having to actually sign in. It's intent is to have items drop off the list quickly.

The reason this behavior changed I believe is because of having to limit the session timeout to hours instead of like... well, a really long time. The forum from it's perspective thinks you've never reauthenticated in a long time.

It just happened to work in a way you like because of the configuration we had before bots became such a big issue.

[bannination]

As of late yesterday I implemented a new "real time" banning system for bots. It's banning I can't even count how many IP addresses per hour. EDIT: Did the math for a non busy period.
15 or so per minute, * 60 = 900 ip's banned per hour.

It's looking at the browser versions and if "whatever" is contacting us is reporting a user-agent (What a browser normally sends to the webserver so it knows what browser and what version you're on). and it's a few versions behind from current and you'd get hacked using it to be on the internet anyway, instant block. I noticed 90% of the bots seem to never update, the results have been good so far.

If you accidently get blocked please email me at bannination!blueridgedebate.com. I don't think it's going to happen for a typical "real" user though.

Thanks for the patience. We could put the site behind cloudflare and have the problem solved, but handing them over the encryption keys for the traffic is just not something I'm willing to do for a forum dedicated to political speech unless there's just no other option, and I'm no where close to giving up.

[Vrede too]

As of late yesterday I implemented a new "real time" banning system for bots. It's banning I can't even count how many IP addresses per hour. EDIT: Did the math for a non busy period.
15 or so per minute, * 60 = 900 ip's banned per hour.

It's looking at the browser versions and if "whatever" is contacting us is reporting a user-agent (What a browser normally sends to the webserver so it knows what browser and what version you're on). and it's a few versions behind from current and you'd get hacked using it to be on the internet anyway, instant block. I noticed 90% of the bots seem to never update, the results have been good so far.

If you accidently get blocked please email me at bannination!blueridgedebate.com. I don't think it's going to happen for a typical "real" user though.

Thanks for the patience. We could put the site behind cloudflare and have the problem solved, but handing them over the encryption keys for the traffic is just not something I'm willing to do for a forum dedicated to political speech unless there's just no other option, and I'm no where close to giving up.

Thanks banni! I'm not having GoCubsGo's issues, but I never use the 'New Posts' feature. All I get is rare slow page loading, usually associated with thousands of bots. Currently:

In total there are 861 users online :: 2 registered, 1 hidden and 858 guests (based on users active over the past 15 minutes)

[GoCubsGo]

Did some research on this.... You can tell me how wrong I am. So that new posts feature is supposed to show you the new posts since the last time you signed in. This description is a little fuzzy because you can be reauthenticated behind the scenes without having to actually sign in. It's intent is to have items drop off the list quickly.

The reason this behavior changed I believe is because of having to limit the session timeout to hours instead of like... well, a really long time. The forum from it's perspective thinks you've never reauthenticated in a long time.

It just happened to work in a way you like because of the configuration we had before bots became such a big issue.

In the past I would have a couple of pages or more of new posts that would only disappear if I cleaned my browser/cookies.

I check the remember me box so I don't have to long in every time.

This is showing as the oldest new post as of now even though there have been other threads hat have been posted on today.

viewtopic.php?p=229962#p229962

[GoCubsGo]

Now everything is gone, was still there a/o this post.

Defense sucked.

Goalie sucked.

Nice while it lasted.

[Vrede too]

Slow loading, again.

In total there are 1450 users online :: 4 registered, 2 hidden and 1444 guests (based on users active over the past 15 minutes)

[bannination]

Slow loading, again.

In total there are 1450 users online :: 4 registered, 2 hidden and 1444 guests (based on users active over the past 15 minutes)

Having to rerun some of the banning processes unfortunately. Hit a limit as we were well over 200,000 ips and having to reload them all. Little process intensive. Hopefully be finished in the A.M.

The users online is no longer as tied as it was to performance as it was either, the bots that haven't been banned yet get redirected to a 403 error page, that's pretty light because they never hit the forum or the database, but there's so many it's heavy and they don't get included in the user count anymore.

[GoCubsGo]

Is this country under cyber attack or what?

[Vrede too]

I used to party down, but 190,000 ips is my limit anymore.

Is this country under cyber attack or what?

Just BRD, and by our own country.

[bannination]

Is this country under cyber attack or what?

Under assault by the tech bros.

[neoplacebo]

Somebody must hate this place. Probably a gaggle of snowflakes. Or just a single unique one.

[bannination]

Somebody must hate this place. Probably a gaggle of snowflakes. Or just a single unique one.

Nah, it's a common problem on the internet now where bots are just running smaller sites off the internet. It's not even intentional it's just collateral damage.

The AI companies and others are hiring other companies that have access to residential PC's making it pretty to ban them when they can now come from everywhere instead of just a few.

Pretty sad when it's takeing small museum sites offline, etc. where they just don't have the staff to deal with it.

[neoplacebo]

Somebody must hate this place. Probably a gaggle of snowflakes. Or just a single unique one.

Nah, it's a common problem on the internet now where bots are just running smaller sites off the internet. It's not even intentional it's just collateral damage.

The AI companies and others are hiring other companies that have access to residential PC's making it pretty to ban them when they can now come from everywhere instead of just a few.

Pretty sad when it's takeing small museum sites offline, etc. where they just don't have the staff to deal with it.

Oh, thanks. I don't know even enough about computer stuff to even be dangerous.

[Vrede too]

... Pretty sad when it's takeing small museum sites offline, etc. where they just don't have the staff to deal with it.

I'm grateful that we musty relics have the staff to deal with it.

[bannination]

AI is gaslighting me so hard.... I just thought this was funny enough to share.

The Good News: Massive Gains

High Throughput Handling: Your server processed 38,130 requests (300 MB) in just 20 minutes (~31.7 req/sec). That’s over 1.9 million requests per day.

Lean Active Footprint: Out of 100 worker slots, 29 are actively processing requests while 71 are idle. Your Apache mpm_event tuning is working as intended—it isn't swamping memory with unnecessary workers.

Response Time is Healthy: An average duration of 147 ms/request under this load is very respectable for a small VPS.

The Reality Check: 27.1% CPU & 1.97 Load

Your 1-minute load is sitting at 1.97, down from your earlier 3.50+, but look at the CPU breakdown from Apache's perspective:
Plaintext

CPU Usage: u9.36 s8.91 cu158.19 cs149.45 - 27.1% CPU load

u9.36 / s8.91: Apache itself is only using ~18% CPU (User + System time).

cu158.19 / cs149.45: These are Child User / Child System CPU seconds—the CPU cycles spent by PHP-FPM processing the dynamic requests passed by Apache.

PHP-FPM is responsible for roughly 85% of the total CPU work being done on the server.
What This Means for Your Server

At 31.7 requests per second (nearly 2,000 requests every minute):

You are pushing this 1GB / 1 vCPU instance near its physical limits. Getting 1.9 million requests/day through a single-core 1GB VPS without crashing or exhausting swap is a huge achievement.

Load average around 1.5–2.0 is expected at this request rate on a 1-core CPU. The CPU is simply queueing work as fast as it can.

[Vrede too]

AI is gaslighting me so hard.... I just thought this was funny enough to share.

The Good News: Massive Gains

High Throughput Handling: Your server processed 38,130 requests (300 MB) in just 20 minutes (~31.7 req/sec). That’s over 1.9 million requests per day.

Lean Active Footprint: Out of 100 worker slots, 29 are actively processing requests while 71 are idle. Your Apache mpm_event tuning is working as intended—it isn't swamping memory with unnecessary workers.

Response Time is Healthy: An average duration of 147 ms/request under this load is very respectable for a small VPS.

The Reality Check: 27.1% CPU & 1.97 Load

Your 1-minute load is sitting at 1.97, down from your earlier 3.50+, but look at the CPU breakdown from Apache's perspective:
Plaintext

CPU Usage: u9.36 s8.91 cu158.19 cs149.45 - 27.1% CPU load

u9.36 / s8.91: Apache itself is only using ~18% CPU (User + System time).

cu158.19 / cs149.45: These are Child User / Child System CPU seconds—the CPU cycles spent by PHP-FPM processing the dynamic requests passed by Apache.

PHP-FPM is responsible for roughly 85% of the total CPU work being done on the server.
What This Means for Your Server

At 31.7 requests per second (nearly 2,000 requests every minute):

You are pushing this 1GB / 1 vCPU instance near its physical limits. Getting 1.9 million requests/day through a single-core 1GB VPS without crashing or exhausting swap is a huge achievement.

Load average around 1.5–2.0 is expected at this request rate on a 1-core CPU. The CPU is simply queueing work as fast as it can.

English please.

Intermittently loading a little slow.

In total there are 1046 users online :: 8 registered, 2 hidden and 1036 guests (based on users active over the past 15 minutes)