Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Ahmed Heasat 10 posts 90 karma points
    Sep 16, 2019 @ 16:34
    Ahmed Heasat
    0

    Got System.OutOfMemoryException when migration big data

    Hi,

    We're currently trying to migrate some external data to our Umbraco8 website, we used contentService to achieve that, but we had some issues in the migration and we couldn't migrate all the data.

    We use this code to migrate:

    public class MigrationController : SurfaceController
    {
        public void CreateNode()
        {
                IMediaService ms = Services.MediaService;
                IContentService contentService = Services.ContentService;
                string imagePath = "C:\\Users\\Admin\\Desktop\\Galaxy.jpg";
                string imageName = Path.GetFileName(imagePath);
                var imagesNode = Umbraco.Content(1063);
                List<string> mediaUDIS = new List<string>(1);
                var parentUDI = new GuidUdi("images", imagesNode.Key);
                for (int j = 0; j < 270000; j++)
                {             
                  var contentTypeBaseServiceProvider = Services.ContentTypeBaseServices;
                  IMedia mediaFile = ms.CreateMedia(imageName, 1057, "Image");
                  using (var fs = new FileStream(imagePath, FileMode.Open))
                  {
                    string nodeName = Path.GetFileName(imagePath) + j.ToString();
                    mediaFile.SetValue(contentTypeBaseServiceProvider, "umbracoFile", imageName, fs);
                    mediaUDIS.Add(mediaFile.GetUdi().ToString());
                    ms.Save(mediaFile);
    
                    IContent content = contentService.CreateContent(nodeName, parentUDI, "imageItem");
                    content.SetValue("images", string.Join(",", mediaUDIS.Select(i => i.ToString())).Trim(new char[] { ' ', ',' }));
                    content.SetValue("text", "Lorem Ipsum is simply dummy text of the printing and typesetting industry.");
                    contentService.SaveAndPublish(content); 
                    mediaFile = null;
                    mediaUDIS.Clear();
                  }
                 }
         }
     }
    

    We got this exception :

    > $exception   {{System.ArgumentOutOfRangeException: capacity was 
    less than the current size.
    
    > Parameter name: requiredLength
    at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
    at System.Text.StringBuilder.Append(Char value, Int32 repeatCount)
    at System.Text.StringBuilder.Append(Char value)
    at System.Text.StringBuilder.AppendFormatHelper(IFormatProvider 
    provider, String format, 
    ParamsArray args)
    at System.String.FormatHelper(IFormatProvider provider, String format, 
    ParamsArray args)
    at System.String.Format(IFormatProvider provider, String format, 
    Object[] args)
    at System.Environment.GetResourceString(String key, Object[] values)
    at System.Exception.get_Message()}}  System.OutOfMemoryException
    

    Here is a snapshot of memory usage we have taken after we got the exception :

    Memory Usage

    We noticed that the memory usage of the SQL server is extremely high as shown below :

    Task Manager

    Also we noticed that the Examine files (internal&external) have become 6GB and the Nucache file: 1.3GB

    Please advice in this matter of the best practices in migrating data..

  • SteveV 48 posts 213 karma points
    Sep 16, 2019 @ 20:08
    SteveV
    0

    I'm curious, how many nodes were you able to create before the exception happened?

  • Ahmed Heasat 10 posts 90 karma points
    Sep 17, 2019 @ 07:13
    Ahmed Heasat
    0

    We have almost 70,000 nodes created before the exception happened

Please Sign in or register to post replies

Write your reply to:

Draft