Crawling and Indexing

 Crawling and indexing is done by a computer that searches the web constantly looking for new content to index. Below I will explain a low key version of what is going on. If you want to get more detailed then please look into my post on Search Engine Optimizing.

Think about the web as a long hallway filled with billions of doors branching off this hallway. Each door in this hallway leads to a previous indexed page. Each one of these doors has its own unique URL. The crawler opens these doors and searches the page it enters for other URL's.  Thank of your blog as a big house. The crawler enters through the front door and then search the house entering door after door (post after post). It then travels through other post and on (each containing a different URL) each path it follows it finds (indexes) more post . Each time it records and looks for new content, making updates to the previous indexed pages and indexing the new content. The crawler will only dig so deep before it stops.  This is where good navigation comes in on your part.

The best navigation is being able to reach any post in your blog by no more the 3 clicks from any post in you blog. Meaning that it should not matter what post or where I am at in your blog. I need to be able to reach any other post within your blog by no more then 3 clicks. This is why it is important to make use of Link List gadgets and your Page Gadget.

Lots of people don't  use a Page gadget on their site. This can cause problems for indexing If you don't have some other form of navigation. Most of the time you would have your Page gadget links leading to a list of links in your blog. Each of these links leading to another post. This makes it easy for the crawler to find new content on you blog. Since a Page gadget displays on every post in your blog, the crawler can travel down the path and find all your post when it crawls your blog. So if your not making use of the Page gadget it can cause some of your post not to get indexed as fast or if at all. If you don't want to use a Page gadget them make sure you have a Link List or Archive on your blog. The crawler and your viewers need a way to find what they are looking for. So it is very important to have good navigation on your blog.

Now this is not to say the crawler will not find a post that is not linked to your blog. But if it does it was merely by luck. So don't count on it to happen all the time.