Why Attracting Spiders to Your Website is a Good Thing

By The Black Fin Team


Category: SEO

Imagine millions of tiny spiders crawling all over everything you own.

It’s a creepy image, but when it comes to web indexing, spiders are very, very good things. In fact, you want Google’s spiders crawling all over your web pages as often as possible, and you want them to find lots of great, high-quality content while they are there.

But let’s back up a bit.

When you conduct a search in Google, you are almost immediately presented with a list of relevant websites from around the world. Where does Google get this list, and how is the list ordered? The answer is that Google has an army of “spiders” (also knows as Googlebots) constantly searching the web for new and interesting pages. The spiders – which are really computer programs ­– surf the web much like humans, collecting information about pages and bringing it home to Google to index. This crawling and indexing system makes it possible for Google to keep track of about 40 billion web pages, and to recommend some of these web pages to millions of users looking for answers and information each day.

Still, Google spiders only index about 10 percent of the web. That means that 90 percent of all pages that exist online will never come up in a Google search. That means 90 percent of the web isn’t reaching users, leading to new clients, or improving anyone’s business. In truth, if you aren’t being noticed by Google’s spiders, your pages aren’t being indexed. And if your pages aren’t being indexed, users aren’t finding you by using the world’s most popular search engine.

I bet you are wishing you had some spiders crawling all over your stuff now.

How can you make that happen?

A really great website is crawled several times per day. A not so good website may only be crawled every six months, or sometimes even more rarely. However, there is plenty you can do to attract spiders to your site and make sure your awesome content is reaching readers:

  • Make a formal request to be indexed. If you ask in the right way, Google will send spiders to crawl your site. Go to Google and fill out an Add Your URL Form. This should, in theory, get your site crawled at least once, though it is uncertain what the timeframe is. In addition, you can upload a XML sitemap to Google Webmaster Tools, which gives the spiders a clear map to follow when they visit your site.
  • Spend extra time on titles and tags. Poorly written titles, meta tags, and author tags can sink your website even if the rest of your content is good. Make certain to put relevant keywords in your titles and meta tags. In addition, connect your author tags to your Google+ profile.
  • Remember that pictures don’t say 1,000 words on the web. Spiders aren’t that smart when it comes to images. They can only read and process text. Use <alt> tags to title all of your images, and be sure to give them keyword-rich, relevant names.
  • Know that Google has trouble forgetting past wrongs. If your domain name is associated with black hat marketing tactics, spam, or shading linking practices, the spiders may have trouble forgiving and forgetting. In fact, these troubling practices could get you de-indexed altogether, forever. Know the history of your domain name before you pour money into a redesign or marketing campaign.
  • Be careful when moving domains. Switching domains can be a good move, especially if your old domain was deindexed, but you must do so carefully and correctly in order to please Google. Make sure your “.htaccess file” is configured and optimized correctly to prevent an indexing nightmare.
  • Build links. Spiders follow links just like humans do. Because of this, the more links that point to your website, the more often spiders will make a visit. Organic link building takes time and effort, but the rewards are worth it.
  • Create fresh, unique, and useful content. If you update your website regularly, Google’s spiders will crawl back often to see what’s new. If they are always finding well-written, relevant, focused, and original content, your Page Rank will increase and the spiders will come back regularly to check for more.
  • Track your success. Many people forget one of the most important steps to improving their search engine optimization: keeping track of how your SEO is affecting your rankings and analyzing what is working and what isn’t working. Start by searching for your website on Google and looking at which pages appear. Take the next step by diving in to Google Analytics for an even more in-depth look at your traffic.

Remember that Google spiders are creatures of habit. If they find great content on your site each time they visit, they will keep visiting. Spiders beget spiders, and spiders make sure your awesome, high-quality pages are indexed and ready to be found by your next great client.

Do you feel like your website is hidden in some dark corner of the web? We can help increase your traffic, your visibility, and your conversions. To learn more about our services, and to start seeing some spiders, contact us today by calling (877) 777-0571 or filling out our quick contact form.

Find out how Black Fin can help
with your free consultation now

Client Review


"I cannot be happier with Gerrid Smith and Black Fin, and highly recommend their services."

- Seth Gladstein, Gladstein Law Firm