System.OutOfMemoryException with Azure Blob Storage
We're running Umbraco 12.2.0 on an Azure S2 App Service instance.
We moved our site to Azure a few months ago, and have had issues with the site restarting itself one or more times a day since moving it. Looking at the log, I see a bunch of the following errors:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)
at System.GC.<AllocateUninitializedArray>g__AllocateNewUninitializedArray|66_0[T](Int32 length, Boolean pinned)
at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
at Azure.Storage.LazyLoadingReadOnlyStream`1..ctor(DownloadInternalAsync downloadInternalFunc, GetPropertiesAsync getPropertiesFunc, DownloadTransferValidationOptions transferValidation, Boolean allowModifications, Int64 initialLenght, Int64 position, Nullable`1 bufferSize, PredictEncryptedRangeAdjustment rangePredictionFunc)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.OpenReadInternal(Int64 position, Nullable`1 bufferSize, BlobRequestConditions conditions, Boolean allowModifications, DownloadTransferValidationOptions transferValidationOverride, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.OpenRead(Int64 position, Nullable`1 bufferSize, BlobRequestConditions conditions, CancellationToken cancellationToken)
at Umbraco.StorageProviders.AzureBlob.AzureBlobItemInfo.CreateReadStream()
at Microsoft.AspNetCore.Http.SendFileResponseExtensions.SendFileAsyncCore(HttpResponse response, IFileInfo file, Int64 offset, Nullable`1 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.StaticFiles.StaticFileContext.SendAsync()
at Microsoft.AspNetCore.StaticFiles.StaticFileContext.ServeStaticFile(HttpContext context, RequestDelegate next)
at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 114
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|8_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Each time, the request path is for a media item. There doesn't seem to be a lot of consistency to the images, just ones that are loaded a lot.
We use Umbraco.Cloud.StorageProviders.AzureBlob to host our media in Azure Blob Storage. Based on the stack trace, this appears to have to do with that. No other log messages precede these, it just starts happening periodically. The puzzling thing is that the memory usage doesn't look bad. Our S2 instance has 3.5 GB RAM. Looking at "Memory working set - Max" in Azure, it shows at 802 MB at one of the times the errors started, and peaks at 1.1 GB in the past 24 hours, but the site has restarted within that timeframe.
Anyone have an idea what's going on or how we might be able to address this?
We are also seeing occasional "NuCache.Content.db is being used by another process" errors, so may be the actual cause of the site restarts, but I'm awaiting 12.3.0 as it sounds like that may fix the NuCache issue, and I'd still like to resolve this "Out of memory" issue.
I did not find a solution. I looked in my logs just now and it appears that the issue hasn't happened recently, I'm not sure why. If you do find a solution please let me know.
I'm hoping it will be resolved when we upgrade to Umbraco 13 anyway, though.
I have no idea if it's related, but if this is an older project that is updated to a newer version of Umbraco, beware that the cache for ImageSharp has been moved to a different NuGet package (Umbraco.StorageProviders.AzureBlob.ImageSharp). Make sure you have that package as well.
When i was profiling the application locally I saw that ImageSharp was allocating a lot of memory. I believe that it cant handle large images in conjunction with many requests. When it cant process the images, the environment gets overwhelmed and then you have OOM-Ex. When i was looking at the data that was being processed I saw a lot of image-files that was a bit large. The files were uploaded by the editors and I believe that the scenario went like this:
requests for images comes from the client
Umbraco retrieves the image from blob storage
Image Sharp tries to process and add query parameters and cache-key to the image to fit the desired ouput.
Enters a log in umbraco and continues
This takes a toll on the CPU, and if you have 32-bit configured it cannot handle the amount of large files. Change to 64-bit. Yes you will get a larger amount of allocated memory but it still can handle it. When I changed to 64-bit in Azure, the CPU was going at 80-90% but no Out Of Memory Exceptions and the site was doing fine. After about 2 hours the CPU went down to about 15-20%. Now its simmering about 10-15%. Memory was stable at about 1000 MB It could be due to less traffic but a stress test passed through whiteout a problem.
But I still believe that image sharp in combination with the storage provider in Umbraco 11 has some issues, but this solved it for me, this far. So the solution would be to reduce the file-sizes and set 64-bit.
We are in the process to upgrade to Umbraco 13, but its nice to have some breathing room
I will report back as soon as business opens and give you guys a report.
Thanks a lot Fredrick!
Your feedback comes in the exact moment for me, as we were struggling with the same issues and wondering if going 64 bit would help. Will definitely try that.
System.OutOfMemoryException with Azure Blob Storage
We're running Umbraco 12.2.0 on an Azure S2 App Service instance.
We moved our site to Azure a few months ago, and have had issues with the site restarting itself one or more times a day since moving it. Looking at the log, I see a bunch of the following errors:
Each time, the request path is for a media item. There doesn't seem to be a lot of consistency to the images, just ones that are loaded a lot.
We use Umbraco.Cloud.StorageProviders.AzureBlob to host our media in Azure Blob Storage. Based on the stack trace, this appears to have to do with that. No other log messages precede these, it just starts happening periodically. The puzzling thing is that the memory usage doesn't look bad. Our S2 instance has 3.5 GB RAM. Looking at "Memory working set - Max" in Azure, it shows at 802 MB at one of the times the errors started, and peaks at 1.1 GB in the past 24 hours, but the site has restarted within that timeframe.
Anyone have an idea what's going on or how we might be able to address this?
We are also seeing occasional "NuCache.Content.db is being used by another process" errors, so may be the actual cause of the site restarts, but I'm awaiting 12.3.0 as it sounds like that may fix the NuCache issue, and I'd still like to resolve this "Out of memory" issue.
Hi David, did you find a solution to this issue? Suddenly we are experiencing the exact same symptoms in an Umbraco v12.3.6 site. Thanks!
Hi,
Did you find a solution to the issue?
We suddenly also are facing the exact same exception and we have our site on Azure connected to Azure Blob.
Regards,
Fredrik
I did not find a solution. I looked in my logs just now and it appears that the issue hasn't happened recently, I'm not sure why. If you do find a solution please let me know.
I'm hoping it will be resolved when we upgrade to Umbraco 13 anyway, though.
I have no idea if it's related, but if this is an older project that is updated to a newer version of Umbraco, beware that the cache for ImageSharp has been moved to a different NuGet package (Umbraco.StorageProviders.AzureBlob.ImageSharp). Make sure you have that package as well.
See: https://docs.umbraco.com/umbraco-cms/v/12.latest/extending/filesystemproviders/azure-blob-storage
(Also: update to Umbraco 13 as soon as possible .NET 7 and Umbraco 12 are about to go out of support).
Hi
I think I have solved it...
When i was profiling the application locally I saw that ImageSharp was allocating a lot of memory. I believe that it cant handle large images in conjunction with many requests. When it cant process the images, the environment gets overwhelmed and then you have OOM-Ex. When i was looking at the data that was being processed I saw a lot of image-files that was a bit large. The files were uploaded by the editors and I believe that the scenario went like this:
This takes a toll on the CPU, and if you have 32-bit configured it cannot handle the amount of large files. Change to 64-bit. Yes you will get a larger amount of allocated memory but it still can handle it. When I changed to 64-bit in Azure, the CPU was going at 80-90% but no Out Of Memory Exceptions and the site was doing fine. After about 2 hours the CPU went down to about 15-20%. Now its simmering about 10-15%. Memory was stable at about 1000 MB It could be due to less traffic but a stress test passed through whiteout a problem.
But I still believe that image sharp in combination with the storage provider in Umbraco 11 has some issues, but this solved it for me, this far. So the solution would be to reduce the file-sizes and set 64-bit.
We are in the process to upgrade to Umbraco 13, but its nice to have some breathing room
I will report back as soon as business opens and give you guys a report.
I hope this helps you guys!
Until tomorrow!
Regards,
Fredrik
Hi again,
I can report that we are out of the woods. The site is performing fine. No exceptions or bottle necks.
Hopes this helps someone
Regards,
Fredrik
Thanks a lot Fredrick! Your feedback comes in the exact moment for me, as we were struggling with the same issues and wondering if going 64 bit would help. Will definitely try that.
is working on a reply...