Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • KB 14 posts 95 karma points
    Dec 30, 2015 @ 12:25
    KB
    0

    umbraco.config keeps recreating itself

    I'm working on project that now has ~50000 nodes and average publish rate is 5-20 nodes per hour from 6 AM till 10 PM. From almost two weeks I'm having big performance issues, the IIS takes 80% of RAM and 80-100% of CPU. All I could find till now is that umbraco.config is being recreated every 30-90 minutes. In log file I found entry that corresponds in time with disappearance of umbraco.config:

    2015-12-30 12:56:36,053 [6] INFO  Umbraco.Core.Publishing.PublishingStrategy - [P3496/T1/D2] Content 'My Article' with Id '80434' has been published.
    2015-12-30 12:56:44,321 [6] INFO  umbraco.content - [P3496/T1/D2] Save Xml to file...
    2015-12-30 12:56:48,143 [6] ERROR umbraco.content - [P3496/T1/D2] Failed to save Xml to file.
    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
       in System.Text.StringBuilder.ToString()
       in umbraco.content.SaveXmlToString(XmlDocument xml)
       in umbraco.content.<SaveXmlToFileAsync>d__b.MoveNext()
    

    At first it happend only during big page load, 2-3 page views per second, now it's happening all around the clock. I've already set up cache on main pages, it helped only for few days. Anyone got a clue where to look?

    My Umbraco version is 7.2.8

  • Nik 1593 posts 7151 karma points MVP 6x c-trib
    Dec 30, 2015 @ 12:32
    Nik
    0

    My first thought is to ask how much RAM does your machine have? and is it a 64bit OS?

    If you can easily increase the RAM, I'd try that first.

  • KB 14 posts 95 karma points
    Dec 30, 2015 @ 12:36
    KB
    0

    Sorry, forgot about it, its Windows Server 2008 R2 Foundation x64 with 8GB RAM and Intel Xeon processor.

    I'm trying to setup a better one but for now 8GB must do...

  • Nik 1593 posts 7151 karma points MVP 6x c-trib
    Dec 30, 2015 @ 15:17
    Nik
    0

    I wouldn't have thought 8GB would be plenty, I kind of expected you to say 4GB or something smaller.

    Does it ever successfully create the XML file?

  • KB 14 posts 95 karma points
    Dec 30, 2015 @ 15:49
    KB
    0

    Yes it does, size 40MB. And it stays for about 30-90 minutes, then it disappears for 30-40 minutes and the same thing again, Failed to save Xml to file -> OutOfMemoryException. Sometimes it won't even reach 100% of RAM and still the same info in log files. I've checked if IIS is running x86, it would explain not reaching 100% of RAM but no, it's running x64.

    I'm wondering when Umbraco can trigger xml recreate and not write to existing one.

  • Nicholas Westby 2054 posts 7100 karma points c-trib
    Dec 30, 2015 @ 16:58
    Nicholas Westby
    0

    Which version of Umbraco? Some older versions had some memory leaks.

    By the way, you can configure IIS to automatically recycle an application pool once the website reaches a certain amount of RAM usage. Not ideal, but could be a workaround for your situation.

  • Nicholas Westby 2054 posts 7100 karma points c-trib
    Dec 30, 2015 @ 17:23
    Nicholas Westby
    0

    Oops, I see that you actually mentioned the Umbraco version (7.2.8). I thought that one had the memory leak fixes.

  • KB 14 posts 95 karma points
    Dec 30, 2015 @ 17:33
    KB
    0

    Ok, I'll try pool recycling, thanks for tip.

  • Charles Afford 1163 posts 1709 karma points
    Dec 30, 2015 @ 20:33
    Charles Afford
    0

    Out of interest are you doing anything custom in your code? Nothing that is doing something on any same events? What surprises me is that you are saving one node and not a whole site publish? Is there anything else that could be causing it like lucene?

  • KB 14 posts 95 karma points
    Dec 30, 2015 @ 21:35
    KB
    0

    Out of interest are you doing anything custom in your code?

    Custom code... there is one event ContentService_Published that I use for three things:

    1. Integer field incrementation
    2. Filling DateTime field for first publish date
    3. Filling text field that stores IDs of documents that are alike

    Two first were here from the start, third one is about two months old and the most resource greedy thing here is:

    var elems = umbraco.cms.businesslogic.Tags.Tag.GetDocumentsWithTags(tags)
                    .Where(n => n.Id != content.Id)
                    .OrderByDescending(n => n.Id)
                    .Take(3);
    

    tags is string and content is node that's being published. When I first added this code, spike in resource consumption was about 2%.

    And there's the second event ContentService_Saving used for logging IP adresses.

    Nothing that is doing something on any same events? What surprises me is that you are saving one node and not a whole site publish?

    I'm saving only one node because there can be up to 3 persons working at the same time on content but on different nodes.

    Is there anything else that could be causing it like lucene?

    I'm using Lucene for search page. From IIS logs I can see that it's being used 100-200 times per day. This page and Lucene were here almost from the start. Can Lucene indexing be so load heavy?

  • Charles Afford 1163 posts 1709 karma points
    Dec 30, 2015 @ 22:23
    Charles Afford
    0

    Have you got the code for those events? I am wondering if its in a foreach loop or something? Can we see the ContentService_Saving code? Where is it logging them to?

    Sorry what i meant by "what surprises me" is that if it was a memory leak i would expect you to publishing 30000 nodes not 2. Which makes me think it probably is not that. And the fact Nicholas thinks it has been patched.

    It could only be a problem if you were saving content manually i guess.

    Do you have anything custom on the save events? Its odd it gone from part time to full time.

    This seems to be in the same ball park.

    http://stackoverflow.com/questions/31270330/umbraco-log-error-failed-to-load-xml-from-file

  • KB 14 posts 95 karma points
    Jan 02, 2016 @ 23:14
    KB
    0

    Here are the triggers on Saving and Published:

    protected override void ApplicationStarting(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
        {
            ContentService.Saving += ContentService_Saving;
            ContentService.Published += ContentService_Published;
        }
    
        void ContentService_Saving(IContentService sender, Umbraco.Core.Events.SaveEventArgs<IContent> e)
        {
            foreach (var entity in e.SavedEntities)
            {
                if (entity.ContentType.Alias == "uCommentsyComment")
                {
                    string ip = System.Web.HttpContext.Current.Request.UserHostAddress;
                    if (ip != null)
                    {
                        var ipfieldvalue = entity.GetValue("ipaddress");
                        if(ipfieldvalue == null || ipfieldvalue == "")
                        {
                            long ipnum = (long) (uint) IPAddress.NetworkToHostOrder((int) IPAddress.Parse(ip).Address);
                            entity.SetValue("ipaddress", ipnum.ToString());
                            IServiceProvider provider = (IServiceProvider)HttpContext.Current;
                            HttpWorkerRequest worker = (HttpWorkerRequest)provider.GetService(typeof(HttpWorkerRequest));
                            int remotePort = worker.GetRemotePort();
                            entity.SetValue("port", remotePort.ToString());
                        }
                        string dat = entity.GetValue("uCommentsyCommentDate").ToString();
                        System.DateTime dt = System.DateTime.Now;
                        if(System.DateTime.TryParse(dat, out dt))
                        {
                            dat = dt.ToString("dd-MM-yyyy HH:MM:ss");
                        }
                        string naz = entity.GetValue("uCommentsyCommentName").ToString();
                        entity.Name = "Comment "+ dat;
                    }
                }
            }
        }
    
        void ContentService_Published(Umbraco.Core.Publishing.IPublishingStrategy sender, Umbraco.Core.Events.PublishEventArgs<Umbraco.Core.Models.IContent> e)
        {
            const string publishedDateKey = "publishdate";
            const string commentcount = "commentcount";
            const string alike = "alike";
            var contentService = ApplicationContext.Current.Services.ContentService;
    
            foreach (var content in e.PublishedEntities.Where(x => x.HasProperty(publishedDateKey)))
            {
                var existingValue = content.GetValue(publishedDateKey);
                if (existingValue == null && content.ParentId != 2109)
                {
                    content.SetValue(publishedDateKey, DateTime.Now);
                    contentService.SaveAndPublishWithStatus(content, raiseEvents: false);
                }
            }
    
            foreach (var content in e.PublishedEntities.Where(x => x.HasProperty(alike)))
            {
                var existingValue = content.GetValue(alike);
                if (existingValue == null)
                {
                    string tag = content.GetValue("tags").ToString().ToLower()
                        .Replace("city name", "")
                        .Replace("country name", "");               
                    var elems = umbraco.cms.businesslogic.Tags.Tag.GetDocumentsWithTags(tag)
                        .Where(n => n.Id != content.Id)
                        .OrderByDescending(n => n.Id)
                        .Take(3);
                    string ids = "";
                    foreach (var item in elems)
                    {
                        ids += item.Id.ToString() + ",";
                    }
                    if (ids.Length > 0)
                    {
                        ids = ids.Remove(ids.Length - 1, 1);
                        if (content.ParentId != 2109)
                        {
                            content.SetValue(alike, ids);
                            contentService.SaveAndPublishWithStatus(content, raiseEvents: false);
                        }
                    }
                }
            }
    
            foreach (var content in e.PublishedEntities.Where(x => x.ContentType.Alias == "uCommentsyComment"))
            {
                var article = content.Parent().Parent();
                int counter = 0;
                if (article.HasProperty(commentcount))
                {
                    if(article.GetValue(commentcount) != null)
                        counter = (int)article.GetValue(commentcount);
    
                    counter++;
                    article.SetValue(commentcount, counter);
                    contentService.SaveAndPublishWithStatus(article, raiseEvents: false);
                }
            }
        }
    

    For last two days I could turn off my main domain for half an hour and do some test. When it was off the load was almost none... I've tried different publish scenarios and load was acceptable. I'm starting to think that publishing is not the problem.

  • Charles Afford 1163 posts 1709 karma points
    Jan 02, 2016 @ 23:30
    Charles Afford
    0

    I don't think its a problem with Umbraco. I do think its related to publishing. My gut is that the code you have in this publish event is trying to save some value in some format to the node and failing, thus giving you the xml error. I will have a look at the code tomorrow. Might be worth trying to get a member of core team to look at it and see if they have come across the issue before.

    Have you debugged the code?

  • KB 14 posts 95 karma points
    Jan 04, 2016 @ 13:45
    KB
    0

    Could not debug on server but I did some logging, checked times for my code and all were below 1s, during big load it reached 5-8s.

    I think I found the problem when started to turn off places where plugins - packages were used. Due to our client requirement my team had to use uCommentsy. When I turned off part that displays comments the server load dropped by 60%, I don't know yet if the recreating of umbraco.config stopped because it can only turned off for few minutes. I'm betting that inside it are old scripts that don't work good with my Umbraco version. Now I'll probably write my own comments mechanism. After farther testing I'll write if it helped.

  • Dave Woestenborghs 3504 posts 12133 karma points MVP 8x admin c-trib
    Jan 05, 2016 @ 13:48
    Dave Woestenborghs
    0

    Could it be your content published event ?

    I see you are republishing the content item again.

    Maybe this is causing the publishing to end up in a loop ?

    Dave

  • KB 14 posts 95 karma points
    Jan 07, 2016 @ 08:45
    KB
    0

    Republishing was there from the start, it is of course an error, I will change it to Publishing event, thanks for noticing.

  • KB 14 posts 95 karma points
    Jan 11, 2016 @ 12:20
    KB
    101

    And everything started to work, problem was with uCommentsy submit form. Made it from scratch and now it works fine. For future my team will replace whole uCommentsy with our own solution. Thanks for help everyone.

  • Charles Afford 1163 posts 1709 karma points
    Jan 11, 2016 @ 12:52
    Charles Afford
    0

    That makes sense. I never got on well with that package when I used it. Always used Discus. Glad you got it Fixed. What was it with the package. Did you reinstall or just created a new form?

    Can you marked this as solved please :)

  • KB 14 posts 95 karma points
    Jan 11, 2016 @ 13:14
    KB
    0

    I've downloaded the source for uCommentsy and started to trace which line could couse it but didn't find it. My best gues is that it opened for each document new session or meybe it looped on something and used all RAM.

    The solution was to make my own form and use submit method from uCommentsy.

    I've tried Disqus and could not use it on this project because of localizations and some rules that user must confirn that he/she will obey.

  • Nicholas Westby 2054 posts 7100 karma points c-trib
    Jan 11, 2016 @ 16:03
    Nicholas Westby
    0

    Replacing uCommentsy is a good idea. And if you still have the uCommentsy dashboard that you see when you log in enabled, you may want to disable it: https://our.umbraco.org/projects/website-utilities/ucommentsy/general/56336-uCommentsy-Comments-Dashboard-Unusably-Slow

Please Sign in or register to post replies

Write your reply to:

Draft