problems getting website indexed by google

Go to solution

Press Ctrl / CMD + C to copy this to your clipboard.

Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at

Henrik Sunesen 84 posts 281 karma points

Apr 23, 2017 @ 12:52
0

Problems getting website indexed by google
I know that this is maybe not at Umbraco related question, but still hope that someone can help me solve my problem:

I have a new website that has been online over 1 month, it's on a new domain and a completely new site. I have a sitemap.

Under the indexing menu in search console:

Sorry for the picture is in danish, but I tried to translate the most important words.

My question is, is it possible that everything is at it should be and I just have to wait? or are there something wrong with my setting?

My robot.txt looks like this:
```
User-agent: *
Disallow: /bin/
Disallow: /Config/
Disallow: /App_Browser/
Disallow: /App_Code/
Disallow: /App_Data/
Disallow: /App_Plugins/
Disallow: /Controller/
Disallow: /umbraco/
Disallow: /umbraco_client/

Allow: /
Allow: /sitemap.htm
Sitemap: https://www.domainname.com/sitemap.xml
```
Copy Link
Jan Skovgaard 11280 posts 23678 karma points MVP 11x admin c-trib

Apr 24, 2017 @ 17:26

0

Hi Henrik

Are you the developer of the site or have you had a company develop the site for you?

The robots.txt looks fine to me but I'm suspicious that perhaps noindex,nofollow have been set on the robots meta tag. It's usually set to these values when the site is under development (If the developers do it somehow either directly in the code or from a setting in the Umbraco backoffice depending on how they implement things).

So to avoid the site being indexed on a temporary development domain it's set like this

But when you're on the live domain it should be set to index,follow like this

You can check the page source by right clicking on a page on your website and choose the "View source" option and then you can look for the meta tag and see what values have been assigned (Sorry if you're a developer who knows everything about this :))

If the meta tag does not exist then it's the same as setting index,follow and then this is not the issue.

Hope this might help.

/Jan

Copy Link
Henrik Sunesen 84 posts 281 karma points

Apr 24, 2017 @ 17:32
100
Hello Jan,

Thanks for the reply, I think I found the problem:
```
 <httpProtocol>
  <customHeaders>
    <remove name="X-Powered-By" />
           <add name="X-Robots-Tag" value="noindex" />
  </customHeaders>
</httpProtocol>
```
in the web.config, that's the reason why i could see the no-index tag in the "View-Source", I have tried to ask the googlebot to crawl the site again. But i think the problem is gone now :)

Once again thank you.

/Henrik
Copy Link
Jan Skovgaard 11280 posts 23678 karma points MVP 11x admin c-trib

Apr 24, 2017 @ 18:02

1

Hi Henrik

Happy to hear I was able to lead you on the (assumably) right track - I did not even consider this scenario but will make a mental note about this too if I see others having the same issue :)

Happy indexing ;-)

/Jan

Copy Link
is working on a reply...

Please Sign in or register to post replies

Flag this post as spam?