I came across a strange problem while I was reviewing webmaster tools. I signed in with my google account and started reviewing my settings. I chose the website I validated before and clicked Site configuration>Crawler access respectively.

In the Test robots.txt tab, I saw interesting errors that was saying:

Line1:  ?User-agent: *              Syntax not understood
Line2:  Sitemap: Valid Sitemap reference detected
Line3:                                        No user-agent specified

I checked my robots.txt file again and again. There was no question mark in Line1 of the robots.txt file which google claims there is.

I create a new robots.txt file by using notepad in Vista Home Premium operating system and uploaded this new file into my production server. I waited for about a day and checked again. The same error was still there.

I Googled for this and found a link which was talking about exactly the same issue I was having. After creating my new robots.txt file by using ANSI encoding type and checking it from this link problem solved. Well at least seems like that. I have been checking Google and it is still showing me the same error. I think it is going to take some time to reflect new changes.

This is the link I found the solution for this specific problem: http://www.google.com/support/forum/p/Webmasters/thread?tid=28d012e70d5fcdc8&hl=en

This is the link of the web site which gave me the opportunity to check my robots.txt file: http://www.asymptoticdesign.co.uk/cgi-bin/check-url.pl

You can find the related blog entry in the following link as well. http://www.yasarmurat.com/Blog/19/syntax-not-understood-error-for-robots-txt-file-in-google-webmaster-tools.aspx

Have a great day.

-Update:

After waiting some time the change happened as it is expected.