This should be inserted by default indeed. When it isn't included by default then there is probably another robots.txt file on disk. SEOChecker checks for that and doesn't render its own robots.txt file.
When you remove the robost.txt file from disk it should work automatically.
So if I understand this correctly if I have one in the root SEO Checker will leave it alone? If I don't have one SEO Checker will create one with whatever entries it thinks it needs?
Yes that is exactly how it works indeed. There are even some advanced options in case you want to modify it or use it in a multi-domain set-up. That is described in the manual.
Dynamic Robots.txt
I was watching your video on configuring the SEO Checker.
I noticed that when the Robots.txt was previewed the sitemap.xml line has been placed in the file.
I've got the 'Enable XmlSitemap' and the 'Enable dynamic robots.txt file' options switched on but the Robots.txt file does not contain this line.
Do I need to add it manually?
If so, what does the 'Enable dynamic robots.txt file' option really do for you?
Regards, Matt
Hi Matt,
This should be inserted by default indeed. When it isn't included by default then there is probably another robots.txt file on disk. SEOChecker checks for that and doesn't render its own robots.txt file.
When you remove the robost.txt file from disk it should work automatically.
Hope this helps,
Richard
Ah, yes. I do already have one at the root.
So if I understand this correctly if I have one in the root SEO Checker will leave it alone?
If I don't have one SEO Checker will create one with whatever entries it thinks it needs?
Regards, Matt
Hi Matt,
Yes that is exactly how it works indeed. There are even some advanced options in case you want to modify it or use it in a multi-domain set-up. That is described in the manual.
Best,
Richard
Yes, would have helped if I had read that bit in the documentation which explains it all.
Thanks, Matt
is working on a reply...