Tech Leader Pro podcast 2023 week 38, analysis of a bot attack on greppr

This week my web search engine greppr came under a bot attack, in this episode I will discuss what happened and how I fixed it. Notes: I am running late this week due to exhaustion. Last week, my web search engine greppr.org was attacked by a bot. Initially I thought it was real traffic: I seen search queries coming in from multiple IPs, that on first impressions looked like real traffic. However the volume was huge! At least, huge for my fledgling service that basically nobody is using. On September 18th, I seen 30K additional search queries, and a further 25K on the 19th. I got excited! On closer examination however, I could see patterns in the search queries: lots of weird searches for maid and cleaning services, from random IPs and user agent strings, all seconds apart from apparently different users. Why did they do this? Perhaps they think they can inject this content into my site? The fix: dynamic query string parameters that change daily, using an unpredictable pattern. Now, the bot gets 404 responses (it is still running). Good bots should identify themselves via the user-agent header in HTTP requests: this is what the web crawler Greppr uses does, along with the bots from big search engines like Google and Bing. Web security remains an arms race. What I am working on this week: Blog: "Choose your mentors wisely" - https://techleader.pro/a/614-Choose-your-mentors-wisely Media I am enjoying this week: Diaspora by Greg Egan. Notes and subscription links are here: https://techleader.pro/a/615-Tech-Leader-Pro-podcast-2023-week-38,-analysis-of-a-bot-attack-on-greppr