Information wants to be free and one of the most horrendous atrocities on the Internet are sites that restrict their content members-only. Heavy surfers have quite likely come across a search result, clicked the link to visit the website, only to be hit face-first with some variance of denial. At best, you are able to view an excerpt of the content in question but the severity of the restrictions can escalate quite quickly. Some sites require a free account registration, which is rather annoying and pervasive if you think about it. However, the worst offenders actually require a paid membership account before any of the information is accessible!

At this point you might be wondering how search engines like Google are able to scrape web content, index it, and serve it in search results yet it remains inaccessible to you? The majority of pay-wall websites actually detect search bots when they access the site and serve their  full articles for indexing purposes. How do the websites know when it’s Google or you? The simplest manner is every access to a website sends a User Agent string that aids in identify where or what the connection is. Your browser typically would identify itself as such – this is the main way browser usage is tracked across the Internet. Google, of course, would identify itself differently – and that’s what the website is keying on. While it’s possible to change the User Agent that your browser reports, it’s a rather drastic approach for situations that are typically one-off in nature.

BeTheBot is a free service that can be utilized to send webpage requests on your behalf, but instead masquerade as a Google (or Yahoo) search bot. The end result is a removal of the members-only or pay-wall and a clean view of the content you are actually seeking. The process couldn’t be more simple – just copy and paste the restricted link into the web site submission form and BeTheBot handles the rest.

The Washington Post is one of the most well known offenders of this practice. Accessing any article off their website results in a membership login form.

Instead, we can right-click on the link in question and copy it to your clipboard.

Paste the link into BeTheBot’s website.

Finally, a few moments later the content will load in a frame within BeTheBot’s website!

While BeTheBot is not perfect, it’s most definitely worth a bookmarking for when the need arises.