10 Tips for a Spider Friendly Website

What exactly is a spider? Electronic spiders, also known as bots or crawlers, are how the search engines find your site. These spiders are sent out on a mission to find information on the millions of websites that exist. These spiders crawl through the web making lists of words found on websites. These words are the key to people finding the information for which they search.

The spiders will begin crawling through the most popular of websites. They will then begin building their list of keywords and will extend their search by following every link on each site. If there is a link to your site anywhere on the web, the spiders will find it.

Each search engine will have its own rules and standards but the process is basically the same. The spiders, however, can only read textual content and cannot click buttons or use drop down menus. They have limited power and capabilities.

Once the spiders retrieve the information, it is collected in a database which is how the search engines determine your ranking. This information is what people find when they use keywords to search for websites.

So with this in mind, you want your site to be spider friendly so they visit you often. There are things you can do to invite the spiders to visit your site.

What the Spiders Like:

1. Robots.txt file – The first thing the spiders will do is look for a robots.txt file. This file will let them know which pages of your site they need not bother with. All sites should have this file even if it is blank because it is the first thing the spiders look for. Not only does the robots.txt file let the spiders know what not to look for, it is like an invitation for them to come into your site.

2. Head Section – Once the spiders find the robots.txt file, they will move onto the head section of your website. Here they will look at your title, meta tags, and keywords. Be sure your title includes descriptive keywords and is not too long. If you want your entire title read keep it at 60 – 70 characters. Your meta tag description will let the spiders know what your site is about. Make it clear, descriptive and keyword rich. This is what will usually show up in the search engines when people do a search. When listing keywords, be sure that these words will also be in your site content. Do your keyword research to find out the best words to use for your site subject.

3. Website Content – Next on the spiders journey is your website content. Be sure to add fresh content to your site regularly. Several times a week would help encourage the spiders to visit often. Within your content, you need to have the keywords and meta tags that are listed in your head section. This is what the spiders will be looking for. They will also look for and follow links within your content.

4. Site Map – Having a site map on your site will enable the spiders to easily navigate through your site and index more of your pages. Your site map should contain only text links to all your pages.

5. Backlinks – Backlinks or inbound links are very important because it elevates the importance of your site in the eyes of the spiders. The number of quality backlinks to your site is greatly considered when ranking your site. You want backlinks from sites that are relevant to your site and have good rankings as well. If you participate in reciprocal linking where you place a website’s link on your site so they place your link on their site, be careful to only link to quality, relevant sites. Linking to sites just to get more inbound links can hurt your rankings more than help.

What the Spiders Do Not Like:

1. Javascript Navigation – Javascript might be a great idea for the design of your website but unfortunately it is invisible to the spiders. If your links are invisible to the spiders, they will be unable to crawl your site and index your pages thus hurting your ranking. To remedy this, either do not use the javascript navigation or add text links somewhere on your page.

2. Doorway Pages – Doorway pages are standalone web pages visible only to the search engine spiders. Doorway pages are useless and irrelevant to human visitors. Because of this, the search engines consider them to be manipulative and could get your site penalized or banned.

3. Frames – I am with the spiders on this one and will immediately leave any site that is built with frames. They are hard to navigate and hard to read for both humans and spiders. The spiders will only be able to navigate your homepage and will not be able to properly index your site. There are ways around this but the best thing, if possible, is to not use frames at all.

4. Images – Images are very important on a website but unfortunately the spiders cannot read images or the text within the image. Not using images could be very damaging to our sites so when you do use images, be sure to add ALT tags. ALT tags are a way of adding text to an image so the spiders can read what the images are about.

5. Flash – Flash can make a website very dynamic and innovative but again the spiders cannot read or understand it so your site will not be indexed as it needs to be. Using some flash on your otherwise html-based site could be considered ok but if your entire site is built using flash, don’t count on any traffic from the search engines.

See your site as the spiders see it by using a free search engine spider simulator. Just enter your URL and see what the spiders see. You, of course, want to make your site pleasing and easy to use for your visitors but remember it is also important to make the spiders happy which in the long run will increase your ranking and make you happy!