| |
|
|
| Webmaster Resources: Search Engine Optimization Information | |
|
|
|
Search Engine Robots - How They Work, What They Do (Part I)
Automated search engine robots, sometimes called "spiders" or "crawlers", are the seekers of web pages. How do they work? What is it they really do? Why are they important? You'd think with all the fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can't do certain things. Robots don't understand frames, Flash movies, images or JavaScript. They can't enter password protected areas and they can't click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation. How Do Search Engine Robots Work? Think of search engine robots as automated data retrieval programs, traveling the web to find information and links. When you submit a web page to a search engine at the "Submit a URL" page, the new URL is added to the robot's queue of websites to visit on its next foray out onto the web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of the reasons why it is important to build your link popularity and to get links from other topical sites back to yours. When arriving at your website, the automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files the robot doesn't need to concern itself with. Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow the links from one page to another. The entire World Wide Web is made up of links, the original idea being that you could follow links from one place to another. This is how robots get around. The "smarts" about indexing pages online comes from the search engine engineers, who devise the methods used to evaluate the information the search engine robots retrieve. When introduced into the search engine database, the information is available for searchers querying the search engine. When a search engine user enters their query into the search engine, there are a number of quick calculations done to make sure that the search engine presents just the right set of results to give their visitor the most relevant response to their query. You can see which pages on your site the search engine robots have visited by looking at your server logs or the results from your log statistics program. Identifying the robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google's "Googlebot"; others are bit more obscure, like Inktomi's "Slurp". Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers. Along with identifying individual robots and counting the number of their visits, the statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. In the resources section of the end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them. How Do They Read The Pages On Your Website? When the search engine robot visits your page, it looks at the visible text on the page, the content of the various tags in your page's source code (title tag, meta tags, etc.), and the hyperlinks on your page. From the words and the links that the robot finds, the search engine decides what your page is about. There are many factors used to figure out what "matters" and each search engine has its own algorithm in order to evaluate and process the information. Depending on how the robot is set up through the search engine, the information is indexed and then delivered to the search engine's database. The information delivered to the databases then becomes part of the search engine and directory ranking process. When the search engine visitor submits their query, the search engine digs through its database to give the final listing that is displayed on the results page. The search engine databases update at varying times. Once you are in the search engine databases, the robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they have the latest info. The number of times you are visited depends on how the search engine sets up its visits, which can vary per search engine. Sometimes visiting robots are unable to access the website they are visiting. If your site is down, or you are experiencing huge amounts of traffic, the robot may not be able to access your site. When this happens, the website may not be re-indexed, depending on the frequency of the robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then. Resources *SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html *Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/ *Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/ *Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/ *Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company. Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved. Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.
Add to these social bookmarking sites: 
MORE RESOURCES:
Google News |
|
|
|
RELATED ARTICLES
Beating the New Google AdWord Rules with Blogs and RSS When Google Adwords first came on scene, it was not immediately apparent to most people how to utilize the system to create wealth, until John Carpenter popularised it with his Google Cash system, a book he wrote some time back.Since then, Google Adwords has become one of the main traffic generation tools for most affiliates. How To Avoid The 3 Biggest Title Tag Mistakes With Search Engines Your title tag is the most important 3 to 12 words on your Web page. It accounts for up to 80% of your rankings on search engines. Keyword Optimization - How To Achieve It The most important thing you can do for your business and your website is develop relevant keywords that are not too competitive yet generate targeted traffic to your website.It's called keyword optimization. DIY SEO Part 1. Wordtracker for keywords. How You Can Easily Compile A Gigantic Keyword List It doesn't matter what kind of marketing you are involved in.Having the most keywords for your market or niche is a surefire way to win over your competition, and that's a fact!The most important place to have thousands of keywords is in your pay-per-click campaigns, as you want to hit as broad an audience as possible, within your market or niche. Search Engine Optimization - Enhancing Web Site Visibility I've had several prospects and clients say to me "I want my web site to come up on top in search engines." And some have been contacted by an SEO company asking for lots of money and claiming that they can guarantee top search engine results, which in many cases is just false promises. How Do I Improve My Web Site Conversion Rate? Part 3 Question 1How do keywords effect your conversion rate in terms of SEO/SEM (search engine optimization/marketing)?Keywords are important for two reasons.Firstly by using the keywords which relate to your reader you get listed by search engines accordingly meaning that people can find you. How to Increase Alexa Ranking of Your Website Alexa toolbar also useful to Browse expired websites database. Many times when we visit any website we got error message that website is not available. SEO: The Good, The Bad And The Ugly I seem to have created quite a stir, on a particular SEO forum recently. In fact, rumor has it, at one point, my article, "Google's Trap, DMOZ's Nap, And Yahoo!'s Crap" was the hottest topic discussed on this particular forum. Search Engine Optimization (SEO) Strategy - Navigating the Dark Waters of Website Promotion Creating a well-designed website is the first step in your internet marketing strategy that must be backed up with techniques designed to drive traffic to the website for successful, long-term results. You wouldn't consider opening a retail store in a major shopping mall without signage and you shouldn't consider having a nice looking website designed without expanding your web presence in order to be found on the internet. Pay-per-click - the Ultimate Tool to Boost Affiliate Sales The old ways are not always the best ways.The traditional model of making money from affiliate products was simple. Recommended Tools When You Put On The SEO Cap For Your Web Site The name of the game is search engine optimization and there are 20 winners and a million losers. It is a massive game that is conducted each and every minute by the top search engines. SEO and Google Indexing - Why It Requires A Complex Blend of Skills If it was easy, everybody would be doing it. Getting a company's name and products, or services, onto the first page of a genuine Google search isn't a trivial piece of work. Advertise Locally Using Search Engines While search engine advertising has been a great advertising medium for businesses capable of or interested in marketing their products and services to a national or international audience, the effectiveness of this type of advertising was limited for businesses interested in advertising to a local market until very recently.For example, a realtor with a web site in Minneapolis is likely interested in advertising on search terms such as "homes for sale" and "sell my home. Your Search Engine Optimization Strategy: Make Love, Not War When it comes to search engine optimization strategy, there are basically two camps - those who view search engines as adversaries to be conquered at any cost and those who regard search engines as partners in their online marketing efforts. Long-time readers of my articles probably already have a good idea of which camp I fall into; however, I believe both approaches can be effective optimization methods. Search Engine Optimization - A Must As you surf the web take a look around at many of the sites you see. Do you notice anything that seems strange? Well, let me point it out to you. Banned By Google And Back Again The date: 29th July 2005. The time: early morning. Getting Honest With The Search Engines Getting Honest With The Search EnginesI spend a lot of time reading newsletters and forum postings about Search Engine Marketing. More often than not, people are asking about methods used to improve Search Engine rankings: what works, what doesn't, and what will get you in trouble. The Hottest Tip On Getting A Web Site Listed Quickly by Yahoo, Inktomi, MSN, FAST and AltaVista Every webmaster knows that the free search engines are a major source of traffic.They also know that applying all the SEO techniques in the world will not in itself get their web pages spidered, indexed and listed. Search Engine Keywords - What Do People Search For? Do you ever wonder how people search for things on the Internet? What if you knew exactly what words they typed when using a search engine? If you're marketing a product or service it's extremely insightful to know what are the most popular search terms relating to whatever you're marketing.The Overture CompanyOverture.
|
| Home | Site Map |
|
| © 2008 | |
