A few weeks ago I started taking a more active role in monitoring the traffic going to my server. I discovered that lots of people were scraping my sites, or in other words, they were writing programs to extract the data off of my sites without actually browsing them in something like Chrome or Firefox. Very rude.
So I started using iptables, a Linux program that lets you configure the kernel firewall, to block IP addresses that were obviously abusing my services.
One of these scrapers was very persistent. They were scraping my ABA Number Lookup site instead of using the very inexpensive API that I provided. As soon as I blocked an IP, a new one started up. I probably would have let them get away with it but their programming was atrocious. Within the space of a few minutes they were looking up the same routing numbers dozens of times instead of looking up unique routing numbers. So I kept blocking their IPs until apparently they ran out, and the scraping stopped.
A few days later I was hanging out with my family when my cell phone starting ringing on my business line. I answered the phone and was greeted by an individual that needed help signing up for the API. I gave him the information he needed, and then he bashfully asked if I could unblock their IP addresses. Ah hah! This was the man that was hammering my server! Turns out he works for a finance-related company on Wall Street and instead of paying the measly $1 per thousand look-ups he was scraping my site.
So now they are using the API like they should have been the whole time, and I’m making an extra $500 per year. Yay!
Moral of the story: Sometimes it pays to check your logs.