Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Per Bolmstedt 65 posts 289 karma points
    Jun 22, 2021 @ 07:23
    Per Bolmstedt
    0

    Umbraco fails to boot even after database connection is warmed up. It seems the failture to boot is "cached"?

    We have an Umbraco installation that takes around a minute to boot. Sometimes Umbraco will successfully boot. But some of the time Umbraco will fail to boot with the "failed to boot" screen and the logs will point to database time-outs as the reason.

    However, even after the database is verifiably warmed up and radiply accessible from Umbraco (using a diagnostic page outside of Umbraco), Umbraco still shows the "failed to boot" screen immediately on page refresh, and the logs will still point to database time-outs as the reason. Only now these time-outs are happening immediately.

    So it seems Umbraco is "caching" the failure to boot, including the reason for boot failure. Is this correct? Is there a way to turn off this "caching" behavior, so that subsequent page refreshs would result in a successful boot?

    If not, is there a way to increase the time-outs during boot, specifically the database time-out? To stop the boot failures occuring in the first place.

    Or is there a different way to try to get Umbraco to boot after a "failed to boot"?

    The Umbraco connection string is verified to work, and sometimes Umbraco will boot after a restart.

    The installation is load-balanced admin/front setup running on Azure Web Apps. We've never had any problems with this type of setup before.

    The same solution works in the production environment. This is the test environment with rougly the same server sizing.

    The only thing special about this Umbraco solution is that it takes long to start, probably because of custom start-up code or similar.

    Umbraco 8.13.

  • AddWeb Solution Pvt. Ltd 109 posts 353 karma points
    Jun 22, 2021 @ 11:31
    AddWeb Solution Pvt. Ltd
    0

    Hello, Check if App_Data folder is included (having Umbraco.Sdf file)

    It seems to be connection string issue (If you will be debugging your application, then you should first exclude app_data folder from your project)

  • Per Bolmstedt 65 posts 289 karma points
    Jun 22, 2021 @ 12:11
    Per Bolmstedt
    0

    Thanks.

    It's not a connection string issue; Umbraco sometimes successfully boots using the same configuration files, but often fails to boot.

    We have a diagnostic page in the solution that uses the Umbraco connection string to query the database, and this diagnostic page can successfully retrieve data as soon as the page can load. But at this point, Umbraco has already failed to boot.

  • Nik 1512 posts 6673 karma points MVP 4x c-trib
    Jun 22, 2021 @ 12:08
    Nik
    100

    Hi Per,

    So, it sounds to me like the time-out probably isn't a connection time out, but a query/processing time out.

    When a site starts up from, it has to rebuild the nuCache and, I think, it rebuilds the Examine indexes as well to ensure that they are fully up to date.

    With sites that have large amounts of content/media/members, this process can reportedly time out. Although in my experience the process is still on going, but it's hogging all the resources while it runs. If you give it half an hour does it eventually start?

    Have you also checked resources utilisation on your db when the timeouts are occurring?

    Thanks

    Nik

  • Per Bolmstedt 65 posts 289 karma points
    Jun 23, 2021 @ 07:25
    Per Bolmstedt
    0

    Thanks.

    The site does not recover after a while.

    The site does not consume any resources once the "Umbraco failed to boot" message appears.

    I'm not sure any database timeouts are actually occurring.

    After the attempted boot (which takes a little over a minute), the site (both /umbraco and /) immediately start returning the "Umbraco failed to boot" message.

    This is what is written (and only this) to the Umbraco log on each attempted page load:

    Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.
    -> Umbraco.Core.Exceptions.BootFailedException: Boot failed.
    -> System.Data.SqlClient.SqlException: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
       at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       ...
       at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.ExecuteScalar() in D:\\a\\1\\s\\src\\Umbraco.Core\\Persistence\\FaultHandling\\RetryDbConnection.cs:line 214
       ...
       at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(PublishedSnapshotServiceOptions options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IProfilingLogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders) in D:\\a\\1\\s\\src\\Umbraco.Web\\PublishedCache\\NuCache\\PublishedSnapshotService.cs:line 149
       ...
       at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\LightInject\\LightInjectContainer.cs:line 111
       at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItem(IFactory factory, Type itemType) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\ComponentCollectionBuilder.cs:line 33
       at Umbraco.Core.Composing.CollectionBuilderBase`3.<>c__DisplayClass10_0.<CreateItems>b__0(Type x) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\CollectionBuilderBase.cs:line 103
       at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
       at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
       at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
       at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateItems(IFactory factory) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\CollectionBuilderBase.cs:line 102
       at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItems(IFactory factory) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\ComponentCollectionBuilder.cs:line 25
       at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateCollection(IFactory factory) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\CollectionBuilderBase.cs:line 120
       at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\LightInject\\LightInjectContainer.cs:line 172
       at DynamicMethod(Object[] )
       at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 3798
       at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 4657
       at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 6169
       at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 4656
       at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 4649
       at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 3856
       at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 3776
       at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 4743
       at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 4705
       at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\\projects\\lightinject\\src\\LightInject\\LightInject.cs:line 3437
       at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\\a\\1\\s\\src\\Umbraco.Core\\Composing\\LightInject\\LightInjectContainer.cs:line 111
       at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory) in D:\\a\\1\\s\\src\\Umbraco.Core\\FactoryExtensions.cs:line 23
       at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\\a\\1\\s\\src\\Umbraco.Core\\Runtime\\CoreRuntime.cs:line 204
    -> System.ComponentModel.Win32Exception: The wait operation timed out
       at Umbraco.Core.Exceptions.BootFailedException.Rethrow(BootFailedException bootFailedException) in D:\\a\\1\\s\\src\\Umbraco.Core\\Exceptions\\BootFailedException.cs:line 80
       at Umbraco.Web.UmbracoInjectedModule.<>c.<Init>b__18_0(Object sender, EventArgs args) in D:\\a\\1\\s\\src\\Umbraco.Web\\UmbracoInjectedModule.cs:line 367
       at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.<>c__DisplayClass285_0.<ExecuteStepImpl>b__0()
       at System.Web.HttpApplication.StepInvoker.Invoke(Action executionStep)
       at System.Web.HttpApplication.StepInvoker.<>c__DisplayClass4_0.<Invoke>b__0()
       at Microsoft.AspNet.TelemetryCorrelation.TelemetryCorrelationHttpModule.OnExecuteRequestStep(HttpContextBase context, Action step)
       at System.Web.HttpApplication.<>c__DisplayClass284_0.<OnExecuteRequestStep>b__0(Action nextStepAction)
       at System.Web.HttpApplication.StepInvoker.Invoke(Action executionStep)
       at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    

    These two lines seem to be the culprit:

    at Umbraco.Core.Exceptions.BootFailedException.Rethrow(BootFailedException bootFailedException) in D:\\a\\1\\s\\src\\Umbraco.Core\\Exceptions\\BootFailedException.cs:line 80
    at Umbraco.Web.UmbracoInjectedModule.<>c.<Init>b__18_0
    

    So it seems that as soon as UmbracoInjectedModule<T>.Init() is called, the boot-failed exception is thrown, and the rest is just old hat. Perhaps it is a module misbehaving? How would I know which module?

  • Per Bolmstedt 65 posts 289 karma points
    Jun 30, 2021 @ 12:14
    Per Bolmstedt
    0

    If the problem is indeed one of the "injected modules", does anyone have any ideas on how to troubleshoot further? Or how to force a re-init, so that the "boot failed" exception stops getting rethrown?

  • Per Bolmstedt 65 posts 289 karma points
    Aug 31, 2021 @ 06:41
    Per Bolmstedt
    0

    We are still facing this same problem. Does anyone have any tips on how to proceed with troubleshooting?

  • Per Bolmstedt 65 posts 289 karma points
    Sep 06, 2021 @ 11:48
    Per Bolmstedt
    0

    We are blaming this on some sort of corruption of the Umbraco database which caused a seemingly infinite loop when booting. So a query/processing timeout on the surface. But definitely something in the database was the problem. We recreated the database from a different installation + uSync.

  • Gareth Evans 139 posts 330 karma points c-trib
    Jan 06, 2022 @ 21:57
    Gareth Evans
    0

    We've seen a similar issue where when the database server is unavailable, boot fails, then even when database is restored/back online, the Umbraco site will not boot until manually restarted (cached error)

    I'm guessing that a BootFailedException is terminal, meaning boot is not re-attempted until a restart.

  • Peter Dolkens 1 post 21 karma points
    Apr 13, 2022 @ 11:01
    Peter Dolkens
    0

    We've also noticed that if there's any hiccup with the SQL connection, Umbraco takes itself down and refuses to come online without a restart.

    You can get into the /umbraco admin area, demonstrating that the website is in fact online, and able to connect to the database to verify your user connection, but the public site itself won't come back online without a restart.

    Here's a log from one of our staging sites, where there's virtually 0 traffic. Slight network hiccup 5 days ago, and the site has been down ever since.

    2022-04-08 16:02:44,469 [P9068/D2/T42] INFO  umbraco.BusinessLogic.Log - Log scrubbed.  Removed all items older than 2022-04-07 16:02:43
     2022-04-08 16:25:50,907 [P9068/D2/T69] ERROR Umbraco.Core.UmbracoApplicationBase - An unhandled exception occurred
    System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
       at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
       at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
       at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
       at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
       at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
       at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
       at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
       at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction(TransactionRequest transactionRequest, String name, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
       at System.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect)
       at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
       at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
       at System.Data.Common.DbConnection.System.Data.IDbConnection.BeginTransaction(IsolationLevel isolationLevel)
       at Umbraco.Core.Persistence.Database.OpenSharedConnection()
       at Umbraco.Core.Persistence.Database.<Query>d__74`1.MoveNext()
       at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
       at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
       at Umbraco.Core.Persistence.Database.Fetch[T](String sql, Object[] args)
       at Umbraco.Core.Persistence.Database.Fetch[T](Sql sql)
       at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions()
       at Umbraco.Core.Sync.DatabaseServerMessenger.Sync()
       at Umbraco.Web.BatchedDatabaseServerMessenger.UmbracoModule_RouteAttempt(Object sender, RoutableAttemptEventArgs e)
       at Umbraco.Web.UmbracoModule.OnRouteAttempt(RoutableAttemptEventArgs args)
       at Umbraco.Web.UmbracoModule.ProcessRequest(HttpContextBase httpContext)
       at Umbraco.Web.UmbracoModule.<Init>b__12_3(Object sender, EventArgs e)
       at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    ClientConnectionId:ffffffff-ffff-ffff-ffff-ffffffffffff
    Error Number:-2,State:0,Class:11
    ClientConnectionId before routing:ffffffff-ffff-ffff-ffff-ffffffffffff
    Routing Destination:ffffffffffff.tr20000.eastus1-a.worker.database.windows.net,11011
     2022-04-08 20:02:45,476 [P9068/D2/T87] INFO  umbraco.BusinessLogic.Log - Log scrubbed.  Removed all items older than 2022-04-07 20:02:44
    
Please Sign in or register to post replies

Write your reply to:

Draft