Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Max 80 posts 437 karma points
    Jan 06, 2020 @ 16:20
    Max
    0

    multisite robots.txt

    I'm trying on my own to create a robots.txt based on a document type with template. If I go to "http://mysite.com/settings/robots-txt" it works. If I go to the URL Alias "http://mysite.com/robots.txt" I get a 404 not found.

    I'm not sure if it's my URL rewrite rule or if it's a conflict with the umbracoUrlAlias. Any assistance is appreciated. Below is my current implementation.

    I added a URL rewrite rule to remove the trailing slash and the umbracoSettings.config requestHandler "addTrailingSlash>false</addTrailingSlash>"

    The document type has one property editor: umbracoUrlAlias with default value of robots.txt

    Here's the template:

    @inherits Umbraco.Web.Mvc.UmbracoViewPage<ContentModels.RobotsTxt>
    @using ContentModels = Umbraco.Web.PublishedModels;
    @{
    Layout = null;
    Response.ContentType = "text/plain";
    var siteMap = Model.AncestorOrSelf<HomePage>().FirstChild<SiteSettings>().FirstChild<XmlSiteMap>();
    }
    /* my robots.txt text goes here */
    

    URL Rewrite Rule:

    <rewrite>
          <rules>
          <rule name="Remove trailing slash" stopProcessing="true">
            <match url="(.*)/$" />
            <conditions>
              <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
              <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
            </conditions>
            <action type="Redirect" redirectType="Permanent" url="{R:1}" />
          </rule>
          </rules>
        </rewrite>
    

    Published URLs:

    enter image description here

  • Yakov Lebski 594 posts 2350 karma points
    Jan 06, 2020 @ 17:04
    Yakov Lebski
    101

    I think it should be more simple

    <rewrite>
      <rules>
    
        <rule name="robots_txt">
          <match url="robots.txt" />
          <action type="Rewrite" url="/robots-txt/" appendQueryString="false" />
        </rule>
      </rules>
    </rewrite>
    
  • Max 80 posts 437 karma points
    Jan 06, 2020 @ 17:27
    Max
    0

    Needed to make it /settings/robots-txt

    And that seems to make it work.

  • lori ryan 239 posts 573 karma points
    Jun 03, 2020 @ 21:45
    lori ryan
    0

    did u find using this approach that google search console could read your robots?

  • This forum is in read-only mode while we transition to the new forum.

    You can continue this topic on the new forum by tapping the "Continue discussion" link below.

Please Sign in or register to post replies