sitemap time out

David Peck 690 posts 1896 karma points c-trib

May 17, 2017 @ 11:16

The sitemap generated by SEO Checker has started timing out. If I extend the timeout the you do eventually get the sitemap with 3000+ urls in it.

Do you believe SEO Checker should be able to produce this sitemap in a timely manner, or should I use something custom to resolve this?

I'm using Umbraco 7.5.7 and SEO Checker 1.9.4.

Copy Link

Dave Woestenborghs 3504 posts 12135 karma points MVP 9x admin c-trib

May 18, 2017 @ 07:09

Hi David,

I just tested this on large site of us with 10000+ content items. The sitemap, although it takes a while to generate it, loads fine.

Do you have perhaps some UrlProviders in place ? I have seen performance issues with the sitemap in the past when we had several UrlProviders in place.

Dave

Copy Link

David Peck 690 posts 1896 karma points c-trib

May 23, 2017 @ 14:32

I do have some UrlProviders. They are pretty quick, but perhaps over several 1000 nodes they create an issue. I'll disable them and check, though I'm not sure what I'll be able to do about it if they are the issue as they are necessary.

Thanks for the idea!

Copy Link

Dave Woestenborghs 3504 posts 12135 karma points MVP 9x admin c-trib

May 24, 2017 @ 12:33

Hi David,

When using UrlProviders it's good to keep in mind that it will go throught them all until it finds a Url when you do .Url on a content item. You can image that when you have a sitemap with 10000+ items this is a lot of calls to UrlProviders, while probably most of your urls are generated by the default url provider.

You problably registered them like this :

UrlProviderResolver.Current.InsertTypeBefore<DefaultUrlProvider, YourUrlProvider>();

What happens when you do .Url in this case is that it will go through all of them before it hits the default provider.

We had performance issues with this in the past and solved it by replacing the DefaultUrlProvider

This is the code of your custom default urlprovider :

public class CustomDefaultProviderUrlProvider : DefaultUrlProvider

{ private List

public CustomDefaultProviderUrlProvider()
    : base(UmbracoConfig.For.UmbracoSettings().RequestHandler)
{
}


public override string GetUrl(UmbracoContext umbracoContext, int id, Uri current, UrlProviderMode mode)
{
    var content = umbracoContext.ContentCache.GetById(id);

    // we don't have a custom provider so just use the default one
    if (content != null && !this.allDoctypesWithCustomUrls.Contains(content.DocumentTypeAlias))
    {
        return base.GetUrl(umbracoContext, id, current, mode);
    }         

    return null;
}


public override IEnumerable<string> GetOtherUrls(UmbracoContext umbracoContext, int id, Uri current)
{
    var content = umbracoContext.ContentCache.GetById(id);

    // we don't have a custom provider so just use the default one
    if (content != null && !this.allDoctypesWithCustomUrls.Contains(content.DocumentTypeAlias))
    {
        return base.GetOtherUrls(umbracoContext, id, current);
    }

    return null;
}

}

It checks we need custom urls for a doctype, and if not it returns the normal url. Otherwise it returns null and our custom url providers will kick in.

We then registered it like this :

 // replace default url provider
            UrlProviderResolver.Current.InsertTypeBefore<DefaultUrlProvider, CustomDefaultProviderUrlProvider>();
            UrlProviderResolver.Current.RemoveType<DefaultUrlProvider>();

            // custom URL providers
            UrlProviderResolver.Current.AddType<YourUrlProvider>();

We saw a drastic improvement in performance when generating things like sitemaps, navigation, overview lists...anything that requires accessing the Url property of content a lot.

Dave

Copy Link

David Peck 690 posts 1896 karma points c-trib

May 24, 2017 @ 14:04

OK that's useful. Thank you.

Copy Link

Richard Soeteman 4049 posts 12922 karma points MVP 2x

May 24, 2017 @ 12:11

I've tested with 10000 items+ But like Dave said maybe url providers cause issues.

Copy Link

Mark Bowser 273 posts 860 karma points c-trib

Jul 31, 2018 @ 14:57

We have a site using SEOChecker v2 (just upgraded from SEOChecker 1.9.x). The sitemap.xml loads extremely slowly on the initial load. We are running into timeout issues. It speeds up after the initial load. There are no UrlProviders on the site unless they are hidden in some package.

Is this a known issue with SEOChecker v2?

Copy Link

Richard Soeteman 4049 posts 12922 karma points MVP 2x

Jul 31, 2018 @ 15:03

Loading depends on the amount of items how much do you have?

Copy Link

Bradley Kronson 63 posts 114 karma points

Jul 10, 2019 @ 12:28

I've written a "hacked" version version of a site map generator which respects some SEOChecker sitemap config, which seems to work faster and does not time out. Sharing for posterity - you need to change the docTypeAlias to the docTypeAlias of your route

@using SEOChecker.Extensions.Umbraco.Document
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage
@{
    Layout = null;
    Response.AddHeader("Content-Type", "text/xml");
    var homePage = Model.Content.AncestorOrSelf("yourWebSiteRootDocTypeAlias");
    var homePageChildren = homePage.Children(x => x.IsVisible() && x.HasValue("umbracoNaviHide") && !x.GetPropertyValue<bool>("umbracoNaviHide")).ToList();
    const string dateTimeFormat = "yyyy-MM-ddTHH:mm:sszzz";

    var seoConfig = SEOChecker.Core.Config.Configuration.DocumentTypeConfiguration;
}
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
        xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
    <url><loc>@homePage.UrlWithDomain()</loc><priority>1.0</priority><lastmod>@homePage.UpdateDate.ToString(dateTimeFormat)</lastmod></url>
    @RenderChildPages(homePageChildren, dateTimeFormat, seoConfig)
</urlset>
@helper RenderChildPages(IList<IPublishedContent> pages, string dateTimeFormat, IList<DocumentTypeConfig> seoConfig)
{
if (pages.Any())
{
    foreach (var item in pages.Where(x => x.IsVisible()))
    {
        var noIndex = item.HasValue("seoRobots") && item.GetPropertyValue<string>("seoRobots").Contains("noindex");
        var item1 = item;
        var config = seoConfig.FirstOrDefault(x => x.DocumentTypeAlias == item1.DocumentTypeAlias);
        var noIndexDocType = false;
        if (config!=null && config.DocumentTypeAlias == item.DocumentTypeAlias)
        {
            noIndexDocType = config.RobotsTxtIndexOption.Contains("noindex");
        }

        if (!noIndex && !noIndexDocType)
        {
                <url><loc>@item.UrlWithDomain() @seoConfig.Count @(config==null)</loc><lastmod>@item.UpdateDate.ToString(dateTimeFormat)</lastmod></url>
                var children = item.Children(x => x.IsVisible() && x.HasValue("umbracoNaviHide") && !x.GetPropertyValue<bool>("umbracoNaviHide")).ToList();
                if (children.Any())
                {
                    @RenderChildPages(children, dateTimeFormat, seoConfig)
                }
            }
        }
    }
}

Copy Link

Richard Soeteman 4049 posts 12922 karma points MVP 2x

Nov 27, 2020 @ 11:36

Found a stupid bug last week that caused this timeout. Rendering of 2000 nodes is just a matter of MS now. Will be included in 2.10, if you can't wait use RC version below.

https://www.dropbox.com/s/eubnvrg46uxha18/seo-checker210_0.zip?dl=0

Copy Link

David Peck 690 posts 1896 karma points c-trib

Nov 27, 2020 @ 13:38

It's good to know that I'm not as stupid as I look. Well done.

Copy Link

is working on a reply...

Flag this post as spam?

Sitemap time out