This blog has been around more than six years, making it ancient in the blogosphere. And, while not popular like the big-boy blogs (I run about 125,000 hits a year with about 1500 RSS subscribers according to FeedBurner), I think I have a reasonable-sized audience for the specialized topic I cover (operations research, of course!). People recognize me at conferences and I get the occasional email (or, more commonly, blog comment) that lets me know that I am appreciated. So I like blogging.
The past few months, however, have been a chore due to the amount of comment spam I get. Of course, I have software to get rid of most the spam automatically (Akismet is what I have installed), since otherwise it would be unbearable. Akismet stopped 5,373 spam comments in the last year. This sounds like a lot but that is way down from the heights a few years ago: Akismet stopped 6,711 spams in the month of March, 2009 alone. Unfortunately, it is letting a lot more spam come through for me to judge: in the past year 619 entries were put through to moderation that I determined were spam. This is a frustrating exercise since I like my readers: if they want to say something, I want them to say it! But comment after comment from places like “Sacremento Cabs” or “Callaway Reviews” saying vaguely on-topic things is a bit hard to take. Sometimes it seems that someone has taken the effort to read the blog post and comments:
“From the communication between the two of you I think I can say that I wish I had a teacher like Mr. X and I wish I had a student like Y.”
came in recently, where Mr. X and Y were previous (legit) commentators. But the URL of the commentator was to some warez site, so I deleted it all. Is this a human posting, or just some pattern matching?
Why the sudden influx? Further checking the logs showed that a number of people (a couple hundred per month) are getting to this blog by searching something like ‘site:.edu inurl:blog “post a comment”‘. Sure enough if I do that search (logging out from google and hoping I get something like a generic search result, I get the following:
Wow! Of all the .edu blogs that include the phrase “Post a Comment”, I come in at number 3! Of course, despite my efforts, Google may still be personalizing the search towards me, but clearly I am showing up pretty high to attract the attention of hundreds of people. Through my diligence and efforts, I have made this blog attractive to the Google algorithms (I do seem to be number 1 for “operations research blog” and some other natural searches). This is a great Search Engine Optimization success!
Or not. Because clearly I am attracting lots of people who have no interest in what I have to say but are rather visiting to figure out how they can manipulate me and the blog for their own non-operations research purposes (I am perfectly happy to be manipulated for operations research purposes!). The sponsored link in the search gives it away: there are companies working hard to get comments, any comments, on blogs (presumably any blogs). How many of those 125,000 hits were really my audience (people in operations research or those who would like to know more about it)? Do I really have an operations research audience at all (beyond Brian, Matt, Laura, and a few others who I know personally)?
I’ll spend time thinking about ways to avoid this aggravation. I’ve already put in NOFOLLOW tags, so there is no SEO value to any URLs that get through. I already warn that if the URL submitted is not on operations research, then the comment will be deleted. I could get rid of URLs completely, but I do see legitimate comments as a way of leading people to interesting places. I could add more CAPCHAs and the like, though I am having trouble with some of those myself, particularly when surfing with a mobile device. Or I can put up with deleting useless comments with inappropriate URLs and just relax about the whole thing. But fundamentally: how can you do Search Engine Optimization to attract those you would like to attract without attracting the attention of the bottom-feeders?
On the off chance that one of the …. shall we say, enterprising souls, has made it through this post, perhaps you can explain in the comments why you add a comment when the topic is of no interest to you. Do you get a dime for every one that makes it through? Are you bored and find this a useful way to pass a quiet afternoon? Are you a program, mindlessly grabbing parts of the post and comments to make yourself look more human? Add a comment: I probably won’t let your URL through but at least we can understand each other a bit better.
Maybe you can ask your readers to leave their opinion instead of their comment. That would throw you out of that unwanted google query that real readers probably never use.
Years ago, I had read this article on human blog spamming.
Don: that would work, unless it turns out that “Leave Your Opinion” is even more searched than “Leave a comment”! Maybe I need to go with “L3ave a c0mment” and see if that will work.
My guess is that some of the sites are simply trying to increase their Google PageRank in doing their own SEO feeding off of your high PageRank (as demonstrated by your 3rd ranking in the search above).
Slightly related, here’s a paper on email-based spam (which is similar to comment spam) — “Click Trajectories: End-to-End Analysis of the Spam Value Chain”
http://cseweb.ucsd.edu/~savage/papers/Oakland11.pdf
I’ll admit I haven’t read through the whole paper, but they did go insofar as to purchase products from the spammers which sounds like a fun research project.
I’m not sure, but I think one motivation for comment spam is just to boost the number off sites linking (however irrelevantly) to their site, to boost their placement in search results.
I read your blog regularly (I even told you so at a conference once), and I completely understand the problem you mention. For instance, I started my website (www.learninglover.com) back in August, and within weeks I was getting comments like “I love what you’re doing”, or “I can relate” and immediately felt all warm and fuzzy inside. Buy then I saw the info about the users and saw that it was spam which brought me back down to earth.
But I think the problem is completely tied up in the mission of the sites. If you want to become the popular kid who gets invited to all the parties, then you’ve also got to deal with the “not so popular” kids who want to hang around you and tag along to your success. You can eliminate some of these stragglers by heuristic methods like the “Please do not enter non-operations research websites”, or like some others do by only allowing registered members post comments, but doing that limits your outreach. And the thing is, the more we find ways to eliminate today’s spam, the industry is working just as hard (if not harder) in trying to find ways to get more spam on these sites.
Hi Michael,
I found your opening comment an interesting read as I spend quite a lot of time trying to figure out Google and how it works.. or doesn’t.
For my sins I came across your blog from a different set of search terms. I have developed an APP for mobile phone networks iPhone and Android and am keen to find out more how to make a more effective use of the technology. To date I have had over 7,000 downloads in four months but how to get the best out of this ever changing technology is a steep learning curve (and an expensive one). [MT: URL deleted]
it’s wonderfull posting. After i applied this posting my site can increase to the first search engine with keyword comfortable garden [MT: URL Deleted]
How about putting a captcha with the comment, e.g. recaptcha? I bet there is a plugin for that.
Ironically, by putting the words in the post, I seem to be even higher ranked on that particular search term (now coming in at 2 and 3). I look forward to seeing the sort of comments I end up getting!
Leo: As I said in the post: “I could add more CAPCHAs and the like, though I am having trouble with some of those myself, particularly when surfing with a mobile device.” And since it looks like a human is putting in the comments, I think they would do the extra step to do the capctcha.
If you’re in charge of your search engine optimization campaign, be careful when you’re doing your job. SEO tactics and mistakes (either malicious or innocent) can affect your current rankings across the board – not just with the page you’re working on. I always tell my clients that contribute to, or handle all of their online marketing, that what you don’t know can definitely hurt you. [URL Deleted]
thanks for posting very good info ..[URL Deleted]
I think you are somewhat harsh in deleting those post. If you are at a place where you can give something to other, I think its not bad in sharing something through your post. I don’t think that the people who are going through your interesting post will be going through all the comments.
Think over it there is nothing bad in giving something. [URL Deleted]
Praveen: I think you don’t understand how many of these there are. I’m putting through those not caught by the spam filter, but there are more than a dozen already caught by the filter. If I let all the off-topic ones through, every post would have dozens on comments, none of which were adding to the conversation.
I had same problem on many blogs. I found “Facebook comments” as really good spam-free solution. [URL Deleted]
I have no helpful comment to provide on spam. I simply wanted to tell you that you do have a silent following. “Do I really have an operations research audience at all (beyond Brian, Matt, Laura, and a few others who I know personally?”
Keep blogging, and good luck with the spam.
I did not see any mention to the possibility of using some sort of authentication, as other blogs do. I had the same issue with a personal blog in the past and it solved my problem. Considering that I’ve started doubting that I’m a human being after failing many chatcha tests, I think that it can be a good solution.
Hello Micheal
I understand the problem, more and more people trying to get good backlinks.
Also I have the problem, I have an good racking shop in the Netherlands.
I Try to get good comments for my products but it is mainly backlinks to sites.
It takes a lot of time to these comments and to filter the good from it. [URL Deleted]
Do you find that most of the comments being left are generic spammy comments that are auto generated by programs? I know the spammers have found ways around it, but perhaps a Captcha field for unapproved commenters would help with this problem?
OR put up a few advertisements and start making money from all the spammers who visit your site. [URL deleted]
Michael, the deal is that although the link to you give to a poster’s URL contains a “nofollow” tag, Google in fact counts these links. Links from older .edu sites are prized. Why? Because Google says so and for no other reason. I think I would try a captcha verification for comments to slow down the automated spam comments. [URL deleted]
Thanks medlaw: that is very helpful. I guess I might have to go to captcha verification, but I am already getting the automated ones with the spam filter: I think it is humans getting through.
And I like Mike’s idea of making money, but that can’t be done with a cmu.edu site: they’d be down on me like a ton of bricks in no time.
I’m guessing that the comment spammers are paid by the comment, from one of the many lame services like you found above, and it doesn’t matter if the blog has nofollow or not. It’s a shotgun approach, obviously.
I’m hoping comment spamming will eventually go the way of keyword stuffing (or whatever obsolete SEO tactic would be a good comparison here), since it’s increasingly being devalued by the number of saavy webmasters who nofollow the links in their comments.
This is my first time i visit here. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! Keep up the good work [URL Deleted]
A lot of the time, they’re paid for each .edu link. So, yep, they’re paid for each comment that gets through. Check out fiverr.com. There are folks there selling a bunch of edu links for $5. I’m guessing most of the folk are from India because, anyway you look at it, $5 is too little money for all the work involved in reading stuff you’re not interested in and then posting a comment so that it (hopefully) gets approved. [URL Deleted]
People are simply trying to build links back to their sites because a link from a .edu domain carries extra weight in Google in terms of building the site’s rankings.
captcha verification is fine but it is possible to purchase captcha services that will still enable automatic postings.
It can be quite a problem. [URL Deleted]
Thanks Norio and Jo. Fiverr is quite informative. Lots of people selling 100 backlinks for $5. It does seem lots of work for a small amount of money.
nice info. thank you for your articles [URL Deleted]
OK, I have added recaptcha. Let’s see if that cuts down on the junk without affecting real users very much.
By the way, in the 2.5 days or so since the post, in addition to the comments above, this post has attracted 32 spam comments caught by Akismet and a further 7 not caught by Akismet that I have not included.
Damn! You get more spam comments than I get page views (total).
Don has a good point. Changing the content on your page would make it more difficult for a search engine to find it with operatives that include “comment.” I can relate, though. I manage several blogs, and it seems like I spend days simply weeding through robotic “spam” comments. Incredibly annoying. [URL deleted]
Tried Disqus?
Michael: I realize you’re frustrated, but don’t forget you really do have an audience interested in OR. Keep up the postings and please don’t let the spam discourage you!
Don’t worry Matthew: I’m posting more than ever at the moment. And I think I have a pretty good idea of how to avoid the spam hassles through captchas, text modifications, and inner peace.
Hi Mike
As a newbie to the blogging world, it is interesting to hear of your issues. I would love to have lots of followers! I’ve had a little Spam commenting, but Akismet is pretty good for the level I get. Keep on blogging! [Ed: will do. And remember your blog is on wordpress not blogspot!]
Michael Trick, recaptcha won’t help you. If you really want to stop spammers you need something more advance like image captcha.
You can google it. [URL Deleted]
You can also read “Sophisticated Targeted Link Spam” which documents the similar problem that a Greek professor (and former General Secretary of Information Systems) faced in his blog.
Hello Mike. Came here to leave a comment with a link obviously, so not really operations research audience.
However, I read the post and I could relate to it as any human being can relate to another human being not understanding something – we all have things out of our focus or interest.
To start with, a “nofollow” tag is not a solution, because it’s not clear whether it really DOES NOT pass link juice. A link from an .edu site can boost a site even though it’s a nofollow one, as experiments show. Not everything Google recommends to public is the way they describe it. So, that to answer why people and bots still come here to spam.
Another things – there are scripts doing blog commenting and there are people. Robots are annoying, but they get cut by spam filters easily; people get through and annoy you as well, but these are people making money using an open way (some actually make a lot :). Of course they are not reflecting too much, if it’s closed it’s closed.
I know how it feels to have a thousand of comments from India or Pakistan saying “thank you this is very informative information”, but hey, these people are happy if they make $50, as that would last their families a while. And again — you leave that loophole for them. You do have an .edu page, .edu pages are VERY authoritative, so the links coming from here are valuable. You are saying your are acquainted with SEO, so you understand what all that means. They’ll be coming here like ants to a dirty kitchen. There are people digging through blogs commenting, there are people selling beer on beaches, there are people deciphering captcha codes all day 🙂 You can either expose yourself and complain, or close and be happy. So should you really be complaining about others if you let a chance for them to do what’s unpleasant to you?
Another thing – you wonder how much of your traffic is not operations research targeting. It’s not a commercial niche so normally you should be getting less, that’s true. Hey, here’s a crazy idea – put a flashy banner on top of your page saying sth like “get dofollow comments from .edu sites” and direct it wherever you want 🙂 Or direct it somewhere with an affiliate link and make a buck, just for the thrill of it! If you can’t fight it, direct it and lead it. Sorry for the “enterprising soul” talk if it offends you 🙂
As for the solution – get some plugin that allows commenting using social network accounts. E.g. “one all social login” you can google it (not my site :). Get APIs for these social networks (LinkedIn included, most academic people should be on it I guess), and then disable regular comments! That’s it, you’ll be getting comments from registered users of those networks and spammers don’t survive there for long.
I’ll round up here. Do not take it close to heart and have a good time!
Peace,
Dzi
P.S. And think about that banner 😉
…didn’t mean all the grammatical mistakes, was typing fast, sorry.
Cheers!
Thank you all for your comments. I’m going to close this down at this point, since the words and discussion here is attracting even more spammers than my other posts. I very much appreciate those who took the time to explain their motivation: it was very useful.
In short, “.edu” sites are very attractive, and people are convinced that links are useful even with a “nofollow” tag. So both robots and people aim to get their links here. Robots are generally removed by Akismet, as are many posts from people. A few get through.
Some suggestions were made:
1) Use Recaptcha. I tried that for a week with little obvious effect on the amount of spam that made it through my filters. People comment-spamming obviously will take the extra few seconds to pass the recaptcha. I, on the other hand, find them a pain, so I removed the recaptcha.
2) Make money off this! Add a banner, advertising, etc. Unfortunately, there is a reason .edu sites are valued: advertising (at least at cmu.edu) is strictly prohibited. If I did this, CMU would be on me very quickly.
3) Change the “Leave a comment” phrase to avoid some searches. Good idea, and I have done so.
4) Narrow the comment window. I was allowing two weeks to give comments. Most of the OR-oriented comments occur in the first 2 days. Narrowing the window should reduce the amount of useless commentary. Will do!
5) Live with it. Perhaps some people here just to leave a useless comment will learn a bit about operations research. And having to handle a few comments that get past the filters is really a “first world problem”.
Again, thanks to all who chose to provide some insight. I note that this post attracted 98 spam comments (not counting some of the ones I included above to illustrate the range of what I get).