I have observed the following behavior whenever our SQL server is rebooted or our web server loses connectivity to the SQL server..
Umbraco enters a persistent state after the BootFailedException is thrown. The only way I've found to remedy this is to restart the application or app pool in IIS. All subsequent HTTP requests to the site have the same error rather than trying to reconnect to the database.
I see some resolved Github issues about throwing this exception rather than going to the install screen. That makes sense for security but the persistent failed boot state is difficult to manage without manual intervention.
I don't recall this being an issue for us with version 7. It has only come up in 8, 9, and 10 for us.
Here's a series of logs that lead up to the boot failure after our SQL server was restarted after Windows updates were applied.
Site is running Umbraco 8.18.0.
{
"@t": "2022-11-30T09:05:20.4268190Z",
"@mt": "Failed (will repeat).",
"@l": "Error",
"@x": "System.Data.SqlClient.SqlException (0x80131904): SHUTDOWN is in progress.\r\nLogin failed for user 'REDACTED'.\r\nCannot continue the execution because the session is in the kill state.\r\nA severe error occurred on the current command. The results, if any, should be discarded.\r\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)\r\n at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)\r\n at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)\r\n at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)\r\n at System.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect)\r\n at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)\r\n at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)\r\n at StackExchange.Profiling.Data.ProfiledDbConnection.BeginDbTransaction(IsolationLevel isolationLevel) in C:\\projects\\dotnet\\src\\MiniProfiler.Shared\\Data\\ProfiledDbConnection.cs:line 138\r\n at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() in D:\\a\\1\\s\\src\\Umbraco.Core\\Sync\\DatabaseServerMessenger.cs:line 263\r\n at Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent.InstructionProcessTask.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Compose\\DatabaseServerRegistrarAndMessengerComponent.cs:line 163\r\nClientConnectionId:1e2c38c0-5875-4bd7-965a-808b6ac254c8\r\nError Number:6005,State:1,Class:14",
"SourceContext": "Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent+InstructionProcessTask",
"ProcessId": 16008,
"ProcessName": "w3wp",
"ThreadId": 387,
"AppDomainId": 2,
"AppDomainAppId": "LMW3SVC78ROOT",
"MachineName": "REDACTED",
"Log4NetLevel": "ERROR"
}
{
"@t": "2022-11-30T09:05:39.4737094Z",
"@mt": "Failed (will repeat).",
"@l": "Error",
"@x": "System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)\r\n at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)\r\n at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\r\n at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)\r\n at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.Open()\r\n at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)\r\n at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() in D:\\a\\1\\s\\src\\Umbraco.Core\\Sync\\DatabaseServerMessenger.cs:line 263\r\n at Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent.InstructionProcessTask.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Compose\\DatabaseServerRegistrarAndMessengerComponent.cs:line 163\r\nClientConnectionId:00000000-0000-0000-0000-000000000000\r\nError Number:-1,State:0,Class:20",
"SourceContext": "Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent+InstructionProcessTask",
"ProcessId": 16008,
"ProcessName": "w3wp",
"ThreadId": 118,
"AppDomainId": 2,
"AppDomainAppId": "LMW3SVC78ROOT",
"MachineName": "REDACTED",
"Log4NetLevel": "ERROR"
}
{
"@t": "2022-11-30T09:05:59.8956668Z",
"@mt": "Failed (will repeat).",
"@l": "Error",
"@x": "System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.) ---> System.ComponentModel.Win32Exception (0x80004005): The remote computer refused the network connection\r\n at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)\r\n at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\r\n at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)\r\n at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.Open()\r\n at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)\r\n at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() in D:\\a\\1\\s\\src\\Umbraco.Core\\Sync\\DatabaseServerMessenger.cs:line 263\r\n at Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent.InstructionProcessTask.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Compose\\DatabaseServerRegistrarAndMessengerComponent.cs:line 163\r\nClientConnectionId:00000000-0000-0000-0000-000000000000\r\nError Number:1225,State:0,Class:20",
"SourceContext": "Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent+InstructionProcessTask",
"ProcessId": 16008,
"ProcessName": "w3wp",
"ThreadId": 144,
"AppDomainId": 2,
"AppDomainAppId": "LMW3SVC78ROOT",
"MachineName": "REDACTED",
"Log4NetLevel": "ERROR"
}
{
"@t": "2022-11-30T09:06:03.1300309Z",
"@mt": "Failed.",
"@l": "Error",
"@x": "System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.) ---> System.ComponentModel.Win32Exception (0x80004005): The remote computer refused the network connection\r\n at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.Open()\r\n at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)\r\n at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n at Umbraco.Core.Persistence.Repositories.Implement.DocumentRepository.HasContentForRelease(DateTime date) in D:\\a\\1\\s\\src\\Umbraco.Core\\Persistence\\Repositories\\Implement\\DocumentRepository.cs:line 1071\r\n at Umbraco.Core.Services.Implement.ContentService.PerformScheduledPublishingRelease(DateTime date, List`1 results, EventMessages evtMsgs, Lazy`1 allLangs) in D:\\a\\1\\s\\src\\Umbraco.Core\\Services\\Implement\\ContentService.cs:line 1467\r\n at Umbraco.Core.Services.Implement.ContentService.PerformScheduledPublish(DateTime date) in D:\\a\\1\\s\\src\\Umbraco.Core\\Services\\Implement\\ContentService.cs:line 1396\r\n at Umbraco.Web.Scheduling.ScheduledPublishing.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Scheduling\\ScheduledPublishing.cs:line 68\r\nClientConnectionId:00000000-0000-0000-0000-000000000000\r\nError Number:1225,State:0,Class:20",
"SourceContext": "Umbraco.Web.Scheduling.ScheduledPublishing",
"ProcessId": 16008,
"ProcessName": "w3wp",
"ThreadId": 125,
"AppDomainId": 2,
"AppDomainAppId": "LMW3SVC78ROOT",
"MachineName": "REDACTED",
"Log4NetLevel": "ERROR"
}
{
"@t": "2022-11-30T12:18:15.6465344Z",
"@mt": "{FailMessage} ({Duration}ms) [Timing {TimingId}]",
"@l": "Error",
"@x": "Umbraco.Core.Exceptions.BootFailedException: A connection string is configured but Umbraco could not connect to the database.\r\n at Umbraco.Core.RuntimeState.DetermineRuntimeLevel(IUmbracoDatabaseFactory databaseFactory) in D:\\a\\1\\s\\src\\Umbraco.Core\\RuntimeState.cs:line 188\r\n at Umbraco.Core.Runtime.CoreRuntime.DetermineRuntimeLevel(IUmbracoDatabaseFactory databaseFactory, IProfilingLogger profilingLogger) in D:\\a\\1\\s\\src\\Umbraco.Core\\Runtime\\CoreRuntime.cs:line 498\r\n at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\\a\\1\\s\\src\\Umbraco.Core\\Runtime\\CoreRuntime.cs:line 186",
"FailMessage": "Boot failed.",
"Duration": 43897,
"TimingId": "b92353c",
"SourceContext": "Umbraco.Core.Runtime.CoreRuntime",
"ProcessId": 4896,
"ProcessName": "w3wp",
"ThreadId": 1,
"AppDomainId": 2,
"AppDomainAppId": "LMW3SVC78ROOT",
"MachineName": "REDACTED",
"Log4NetLevel": "ERROR",
"HttpRequestNumber": 1,
"HttpRequestId": "8cb0f869-f3d5-473b-9a5a-14f2667a7a83"
}
Similarly, I see this each time my server is rebooted. I have to log into the server, go to IIS and restart the website. Sometimes I need to recycle the app pool as well before it starts responding again.
This isn't an issue with standard .NET websites, would very much appreciate a fix.
This isn't actually an issue with Umbraco specifically, it is to do with the way IIS initializes out of process applications (.net core apps in general) on startup and occurs because IIS restarts before SQL has fully started.
Connectionstring is correct but yes indeed, sql is not ready at the moment of appool is raised. But is there any solution to this, maybe loop-check or postpone to connect to sql?
I have the same problem and can't find a solution within Umbraco or core itself. The connection string is not terminated by itself - it is terminated when regular updates are made on the server. At that time, the apppools are raised, but since the first 'ping' to sql is unsuccessful, even when sql is set up, it still shows an error. Then I have to manually recycle the apppools. The sysadmin suggested making a command to recycle all apppools after each update after a certain timeout. It's an ugly hack, but it's worth a try...
Are you able to share any details about your utility? I'm about to go live with an umbraco 14 site and worried I will run into this same issue if the DB server is rebooted overnight.
It seems a powershell script on a scheduled task might work.
How to deal with BootFailedException?
I have observed the following behavior whenever our SQL server is rebooted or our web server loses connectivity to the SQL server..
Umbraco enters a persistent state after the BootFailedException is thrown. The only way I've found to remedy this is to restart the application or app pool in IIS. All subsequent HTTP requests to the site have the same error rather than trying to reconnect to the database.
I see some resolved Github issues about throwing this exception rather than going to the install screen. That makes sense for security but the persistent failed boot state is difficult to manage without manual intervention.
I don't recall this being an issue for us with version 7. It has only come up in 8, 9, and 10 for us.
Here's a series of logs that lead up to the boot failure after our SQL server was restarted after Windows updates were applied.
Site is running Umbraco 8.18.0.
Similarly, I see this each time my server is rebooted. I have to log into the server, go to IIS and restart the website. Sometimes I need to recycle the app pool as well before it starts responding again.
This isn't an issue with standard .NET websites, would very much appreciate a fix.
This isn't actually an issue with Umbraco specifically, it is to do with the way IIS initializes out of process applications (.net core apps in general) on startup and occurs because IIS restarts before SQL has fully started.
from log looks likes wrong connection string or database not avaliable
/#Yakov
Connectionstring is correct but yes indeed, sql is not ready at the moment of appool is raised. But is there any solution to this, maybe loop-check or postpone to connect to sql?
I have the same problem and can't find a solution within Umbraco or core itself. The connection string is not terminated by itself - it is terminated when regular updates are made on the server. At that time, the apppools are raised, but since the first 'ping' to sql is unsuccessful, even when sql is set up, it still shows an error. Then I have to manually recycle the apppools. The sysadmin suggested making a command to recycle all apppools after each update after a certain timeout. It's an ugly hack, but it's worth a try...
Regards
I'm working around this in a similar fashion. We have a monitoring utility set up to poll the site and check for these conditions:
If those conditions are met, the app pool gets restarted by the utility.
FB,
Are you able to share any details about your utility? I'm about to go live with an umbraco 14 site and worried I will run into this same issue if the DB server is rebooted overnight.
It seems a powershell script on a scheduled task might work.
I'm using Site24x7.com to monitor. A scheduled task making an http request with curl or some other library could also work.
is working on a reply...