I have observed the following behavior whenever our SQL server is rebooted or our web server loses connectivity to the SQL server..

Umbraco enters a persistent state after the BootFailedException is thrown. The only way I've found to remedy this is to restart the application or app pool in IIS. All subsequent HTTP requests to the site have the same error rather than trying to reconnect to the database.

I see some resolved Github issues about throwing this exception rather than going to the install screen. That makes sense for security but the persistent failed boot state is difficult to manage without manual intervention.

I don't recall this being an issue for us with version 7. It has only come up in 8, 9, and 10 for us.

Here's a series of logs that lead up to the boot failure after our SQL server was restarted after Windows updates were applied.

Site is running Umbraco 8.18.0.

{
    "@t": "2022-11-30T09:05:20.4268190Z",
    "@mt": "Failed (will repeat).",
    "@l": "Error",
    "@x": "System.Data.SqlClient.SqlException (0x80131904): SHUTDOWN is in progress.\r\nLogin failed for user 'REDACTED'.\r\nCannot continue the execution because the session is in the kill state.\r\nA severe error occurred on the current command.  The results, if any, should be discarded.\r\n   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)\r\n   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)\r\n   at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)\r\n   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)\r\n   at System.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect)\r\n   at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)\r\n   at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)\r\n   at StackExchange.Profiling.Data.ProfiledDbConnection.BeginDbTransaction(IsolationLevel isolationLevel) in C:\\projects\\dotnet\\src\\MiniProfiler.Shared\\Data\\ProfiledDbConnection.cs:line 138\r\n   at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n   at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n   at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() in D:\\a\\1\\s\\src\\Umbraco.Core\\Sync\\DatabaseServerMessenger.cs:line 263\r\n   at Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent.InstructionProcessTask.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Compose\\DatabaseServerRegistrarAndMessengerComponent.cs:line 163\r\nClientConnectionId:1e2c38c0-5875-4bd7-965a-808b6ac254c8\r\nError Number:6005,State:1,Class:14",
    "SourceContext": "Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent+InstructionProcessTask",
    "ProcessId": 16008,
    "ProcessName": "w3wp",
    "ThreadId": 387,
    "AppDomainId": 2,
    "AppDomainAppId": "LMW3SVC78ROOT",
    "MachineName": "REDACTED",
    "Log4NetLevel": "ERROR"
}
{
    "@t": "2022-11-30T09:05:39.4737094Z",
    "@mt": "Failed (will repeat).",
    "@l": "Error",
    "@x": "System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)\r\n   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)\r\n   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\r\n   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)\r\n   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n   at System.Data.SqlClient.SqlConnection.Open()\r\n   at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)\r\n   at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n   at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n   at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() in D:\\a\\1\\s\\src\\Umbraco.Core\\Sync\\DatabaseServerMessenger.cs:line 263\r\n   at Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent.InstructionProcessTask.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Compose\\DatabaseServerRegistrarAndMessengerComponent.cs:line 163\r\nClientConnectionId:00000000-0000-0000-0000-000000000000\r\nError Number:-1,State:0,Class:20",
    "SourceContext": "Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent+InstructionProcessTask",
    "ProcessId": 16008,
    "ProcessName": "w3wp",
    "ThreadId": 118,
    "AppDomainId": 2,
    "AppDomainAppId": "LMW3SVC78ROOT",
    "MachineName": "REDACTED",
    "Log4NetLevel": "ERROR"
}
{
    "@t": "2022-11-30T09:05:59.8956668Z",
    "@mt": "Failed (will repeat).",
    "@l": "Error",
    "@x": "System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.) ---> System.ComponentModel.Win32Exception (0x80004005): The remote computer refused the network connection\r\n   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)\r\n   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\r\n   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)\r\n   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\r\n   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n   at System.Data.SqlClient.SqlConnection.Open()\r\n   at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)\r\n   at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n   at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n   at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() in D:\\a\\1\\s\\src\\Umbraco.Core\\Sync\\DatabaseServerMessenger.cs:line 263\r\n   at Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent.InstructionProcessTask.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Compose\\DatabaseServerRegistrarAndMessengerComponent.cs:line 163\r\nClientConnectionId:00000000-0000-0000-0000-000000000000\r\nError Number:1225,State:0,Class:20",
    "SourceContext": "Umbraco.Web.Compose.DatabaseServerRegistrarAndMessengerComponent+InstructionProcessTask",
    "ProcessId": 16008,
    "ProcessName": "w3wp",
    "ThreadId": 144,
    "AppDomainId": 2,
    "AppDomainAppId": "LMW3SVC78ROOT",
    "MachineName": "REDACTED",
    "Log4NetLevel": "ERROR"
}
{
    "@t": "2022-11-30T09:06:03.1300309Z",
    "@mt": "Failed.",
    "@l": "Error",
    "@x": "System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.) ---> System.ComponentModel.Win32Exception (0x80004005): The remote computer refused the network connection\r\n   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n   at System.Data.SqlClient.SqlConnection.Open()\r\n   at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)\r\n   at NPoco.Database.BeginTransaction(IsolationLevel isolationLevel)\r\n   at Umbraco.Core.Scoping.Scope.get_Database() in D:\\a\\1\\s\\src\\Umbraco.Core\\Scoping\\Scope.cs:line 330\r\n   at Umbraco.Core.Persistence.Repositories.Implement.DocumentRepository.HasContentForRelease(DateTime date) in D:\\a\\1\\s\\src\\Umbraco.Core\\Persistence\\Repositories\\Implement\\DocumentRepository.cs:line 1071\r\n   at Umbraco.Core.Services.Implement.ContentService.PerformScheduledPublishingRelease(DateTime date, List`1 results, EventMessages evtMsgs, Lazy`1 allLangs) in D:\\a\\1\\s\\src\\Umbraco.Core\\Services\\Implement\\ContentService.cs:line 1467\r\n   at Umbraco.Core.Services.Implement.ContentService.PerformScheduledPublish(DateTime date) in D:\\a\\1\\s\\src\\Umbraco.Core\\Services\\Implement\\ContentService.cs:line 1396\r\n   at Umbraco.Web.Scheduling.ScheduledPublishing.PerformRun() in D:\\a\\1\\s\\src\\Umbraco.Web\\Scheduling\\ScheduledPublishing.cs:line 68\r\nClientConnectionId:00000000-0000-0000-0000-000000000000\r\nError Number:1225,State:0,Class:20",
    "SourceContext": "Umbraco.Web.Scheduling.ScheduledPublishing",
    "ProcessId": 16008,
    "ProcessName": "w3wp",
    "ThreadId": 125,
    "AppDomainId": 2,
    "AppDomainAppId": "LMW3SVC78ROOT",
    "MachineName": "REDACTED",
    "Log4NetLevel": "ERROR"
}
{
    "@t": "2022-11-30T12:18:15.6465344Z",
    "@mt": "{FailMessage} ({Duration}ms) [Timing {TimingId}]",
    "@l": "Error",
    "@x": "Umbraco.Core.Exceptions.BootFailedException: A connection string is configured but Umbraco could not connect to the database.\r\n   at Umbraco.Core.RuntimeState.DetermineRuntimeLevel(IUmbracoDatabaseFactory databaseFactory) in D:\\a\\1\\s\\src\\Umbraco.Core\\RuntimeState.cs:line 188\r\n   at Umbraco.Core.Runtime.CoreRuntime.DetermineRuntimeLevel(IUmbracoDatabaseFactory databaseFactory, IProfilingLogger profilingLogger) in D:\\a\\1\\s\\src\\Umbraco.Core\\Runtime\\CoreRuntime.cs:line 498\r\n   at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\\a\\1\\s\\src\\Umbraco.Core\\Runtime\\CoreRuntime.cs:line 186",
    "FailMessage": "Boot failed.",
    "Duration": 43897,
    "TimingId": "b92353c",
    "SourceContext": "Umbraco.Core.Runtime.CoreRuntime",
    "ProcessId": 4896,
    "ProcessName": "w3wp",
    "ThreadId": 1,
    "AppDomainId": 2,
    "AppDomainAppId": "LMW3SVC78ROOT",
    "MachineName": "REDACTED",
    "Log4NetLevel": "ERROR",
    "HttpRequestNumber": 1,
    "HttpRequestId": "8cb0f869-f3d5-473b-9a5a-14f2667a7a83"
}