How do search engines find you?

How do search engines 'know' when sites have been added to the web?

Little bits of software known as "spiders" or "crawlers" constantly travel the web. They find websites and send information about them to the search engine indexes. (Don’t read this if you suffer from arachnophobia, but it’s a bit like spiders crawling around in the back garden looking for goodies to cart back to their den).

As it visits a web page, a spider creates a copy of the page for later indexing by the search engine.

What usually happens is that one of these spiders starts off with a list of known web pages to visit. As it visits each page, it will identify all of the links contained on the page and make a note to ‘crawl’ these sites also.

Eager as they are to find your site, there are several barriers which might make it hard for spiders to find you:

• Your site is nott well connected through multiple links to other sites on the web
• Your site was launched after the spider's last visit
• Your site was temporarily unavailable when the spider attempted to access it
• Your design of the site makes it difficult for the spider to crawl the site

The importance of links

Search engine spiders use the links between pages to locate new pages for them to index. So it is very important that pages within your website:

• Are linked to from other websites so that your site can be found in the first place. Generally, getting your website included in web directories is the best place to start in getting your site 'linked into' the web.
• Link to one another so that all pages on your site can be found and indexed. Every page on your site should be reachable.

As discussed in "How do search engines place a website in their rankings" links are also very important in how well your web pages are ranked in search engines' organic listings.

Submitting your site to search engines

Although search engines spiders can find your site from other sites that link to it, most search engines will also allow you to manually submit your site to be included in the search engines' indexes. However, in most instances there is no guarantee that manual submission will ensure your site's inclusion in search engine indexes.
 

Web Master tools and site maps

Most of the major search engines, such as Google and Yahoo, include a range of web master tools and resources to assist you in making your site more "search engine friendly." These tools and resources vary between search engines, but can include content and design guidelines and tips. They can also give you feedback on any barriers that crawlers may be experiencing with your site.

Google also provides a facility whereby you can submit a site map of your site for the crawler to use. This is particularly useful in instances where:

• Pages on your site are dynamically created
• Pages on your site would not be easily found by following links
• Your site is new and does not have a lot of other sites linking to it

However, as with manual submission of your site to a search engine, there is no guarantee that using this site map facility will get your site indexed in Google.