Posts Tagged robots.txt

Send Feedbacks on Crawl-Delay in Bing

Bing is looking for input on Crawl-Delay:

In keeping with Bing themes of rapid change and responding to input and feedback from the webmaster community, Bing webmaster teams are working on future feature planning for the Bing webmaster tools and they would like to hear user and webmasters thoughts in a few areas. They starts from Crawl-Delay.

Today, Bing fully support crawl delay which is specified in robots.txt.

Now Bing are considering creating a tool which allows to directly control the frequency, and perhaps even the timing, of msnbot/bingbot.

A few questions from Bing Team:

  1. Do you prefer for us to continue to support crawl-delay in robots.txt?
  2. Would you like to be able to control the crawl rate based on time of day?
  3. What are the ranges of crawl rate you would like to see if you had control over the time of day?
  4. Would you want to turn crawling on/off by time, or just specify different crawl rates based on the time of day?
  5. If you had this ability to directly control the crawl rate through a webmaster tool, how often would you adjust the settings?
  6. If we built this tool, would you like it to control our fetching directly, or would you rather have the tool output the proper robots.txt entries which you then insert into your robots.txt file?

Feel free to send reply to this post on email to bwmc@microsoft.com

( Source : Bing Webmaster Central Blog )

 

Comments off

SEO Tips # 86 – Comment tags and Robots.txt

Tip # 1:

Don’t stuff keywords in comment tags. Use comment tags in the manner in which the tag was designed to be used — to place comments about the page itself. Don’t put keywords in comment tags at all.

Tip # 2 :

Have you ever felt like you were running low on ideas in Wordtracker?

Try using the tool http://www.thesaurus.com to look up multiple new synonyms which you can explore further in Wordtracker.

Tip # 3:

Place a robots.txt file on every domain. The first thing a spider looks for when it hits your Web site is a robots.txt file. If it’s looking for that file, put it on your site! Robot Manager Pro is an excellent tool for creating robots.txt files as well as monitoring spider traffic

Comments off

SEO Tips #37 – Robots.txt file setup

Do you have a robots.txt file set up on your server? Here’s what Google says in their Webmaster Guidelines:

-Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it’s current for your site so that you don’t accidentally block the Googlebot crawler.

Visit http://www.robotstxt.org/wc/faq.html to learn how to instruct robots when they visit your site.

Comments off

%d bloggers like this: