How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag
Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it's a problem that is easily rectified.
Your primary weapon of choice against duplicate content can be found within "The Robot Exclusion Protocol" which has now been adopted by all the major search engines.
There are two ways to control how the search engine spiders index your site.
1. The Robot Exclusion File or "robots.txt" and
2. The Robots < Meta > Tag
The Robots Exclusion File (Robots.txt)
The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.
Creating your robots.txt file
Example 1 Scenario
Example 2 Scenario
Example 3 Scenario
By naming the particular search spider in the "User-agent" you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.
That's all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it's probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don't want Google to cache the page, because the site contains time sensitive information. This can be achieved simply by adding the "noarchive" directive.
What could be simpler!
Although there are other ways of preventing duplicate content from appearing in the Search Engines this is the simplest to implement and all websites should operate either a robots.txt file and or a Robot tag combination.
Should you require further information about our search engine marketing or optimization services please visit us at http://www.e-prominence.co.uk - The search marketing company
Add to these social bookmarking sites:
Why Pay For Performance SEO Is Really Too Good To Be True (Provided You're ... - Search Engine Journal
Notice: Undefined index: TITLE in /var/www/vhosts/harmonyhollow.net/httpdocs/webmaster-resources/seo/inc/rss.inc on line 103
Search Engine Optimization - A Must
As you surf the web take a look around at many of the sites you see. Do you notice anything that seems strange? Well, let me point it out to you.
Google has an Achilles Heal - Will Their Competitors Notice?
Even though Google Revenues continue to soar, the hidden problem that may stifle growth and may even allow Yahoo or MSN to overtake the paid search market in the future lies in two critical phrases: Customer Support, and Customer TrainingApproximately 40% of the small businesses we have surveyed have tried Adwords in the past and failed, and some of them have tried multiple times. In some markets the percentage is closer to 60%.
Search Engine Optimization (SEO) Strategy - Navigating the Dark Waters of Website Promotion
Creating a well-designed website is the first step in your internet marketing strategy that must be backed up with techniques designed to drive traffic to the website for successful, long-term results. You wouldn't consider opening a retail store in a major shopping mall without signage and you shouldn't consider having a nice looking website designed without expanding your web presence in order to be found on the internet.
Link Building Services
In today scenario when we talk about Search Engine Optimization, we also talk about one of the most important aspect of SEO, which is Link Building. But there are different types, aspects and limitations of Link Building, which would be discussed now under1.
7 Search Engine Optimization Mistakes and Solutions
To many websites, webmasters discover that major sources of website traffic come from search engines. Therefore, they are all keen on gaining top search engine placements through search engine optimization.
The 3 Essential Components of a Search Engine Optimization Campaign
Everyday, the Search Engines average 300 MILLION searches. In a recent Forrester Research report 81% of consumers on the Internet find products and services by using the Search Engines.
All about SEO or SFO?
First let's start with definitions:SEO: Search Engine Optimization, SFO: Search Friendly Optimization.These two things are what most webmasters have trouble balancing.
Things You Must Realize When Searching
For the uninitiated, searching for web pages can seem a slow, obscure process. Unless you have a high-speed Internet connection, web pages may seem to take days to load.
Secrets on Website Promotion: How You Can Get a #1 Ranking for Your Website Name Within 30 Days
Launching a new website with enough acceleration to rise above this ever increasing daily din needs some force. It is common to see a website with a different name and various product or service offerings with equally unrelated names.
How to Succeed with the Search Engines
The Cold Hard Facts?..
Surviving the Search Wars - Local Directories
The pursuit of online information has become an increasingly dynamic and competitive marketplace during the past three years. Global heavyweights such as www.
Surviving Googles Aging Delay
Google has always been the search industry's innovator and that's just what Google's aging delay symbolizes, the evolution of search innovation? yet another significant step forward for Google.Google's success as a search engine can undeniably be attributed to its ability to consistently return the most relevant search engine results.
Site Maps: A Force To Be Reckoned With
Another important component of search engine optimization is the use of site maps. If you want visitors -- and search engine spiders -- to find every page on your Web site, a site map can be your biggest ally especially if you have a lot of content on your site (and if you've been reading all the advice on our site, you should know by now that the more content you have the better your chances are for top ranking).
SEO #6: How TO GET Banned by Google!
Yesterday you should have read the fifth course out of 6 courses that will help you get a TOP rank in the search engines and get EXPLOSIVE LASER TARGETED TRAFFIC for Free. Today we move on to course #6 and study how to get banned by google! Please read today's course very carefully and take some time to test what I'm about to tell you on your own webpage.
Get Traffic You Need - Make Your Links Work
So you have built a nice web site with good solid content and now want to start selling your products or services.Search Engine Optimization (SEO) is rapidly becoming a dying art.
Niche Marketing - Why Keyword Research Come First
A good portion of my business involves spending hours and hours using incredibly powerful but difficult to master software to uncover thousands of the exact, targeted keyword phrases people in any given niche market are typing into the search engines.When people new to the internet discover what I do and see the huge lists of keyword phrases and search data I uncover and compile, they often ask me what the heck it is used for.
Companies Cash In on Your Search Engine Ignorance
This article will cause many companies to stir, but it's about time someone started speaking against these services.It really angers me when I see the numerous services that boast they will increase your traffic by submitting your web site to umpteen different search engines.
SEO: When Being Optimized Can Hurt
It's a marketing dream come true: A potential customer, looking for what you have to offer, types a few words into her favorite search engine and voila! She is led directly to your website where she can go from "prospect" to "customer."The best part is, it didn't cost you anything (except time and elbow grease) to get to the top of her results.
Are You Losing The Battle For Search Engine Traffic?
Search engine traffic should be a priority for any online business and some level of optimization is apart of every effective marketing strategy.On the plus side, search engine traffic is the cream of the crop.
Google Ranking WITHOUT Ever Submitting To Google!
A while back, I read an article that explained how to get a good google rating without ever submitting your site to their submission forms. Like you, I was kind of shocked by this statement so I decided to give it a try.
|Home | Site Map|