Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Duke 11 posts 102 karma points
    Jan 29, 2020 @ 19:23
    Duke
    0

    SQL for v8 Database Migration?

    Are there documented steps and SQL available to manually migrate an SQL database from 7.14.x to 8.x?

    I've been attempting a migration from 7.15.3 to 8.5.1 with very little to show for it and want to go about doing the migration myself without the migration tool.

    Thank you! Jon Portland, Oregon

  • Joep 96 posts 698 karma points
    Jan 30, 2020 @ 08:29
    Joep
    0

    Hi,

    No there isn't really documentation about migrating using SQL only. May I ask what is going wrong with your migration?

    -Joep

  • Steve Megson 151 posts 1022 karma points MVP c-trib
    Jan 30, 2020 @ 09:39
    Steve Megson
    0

    In principle the various schema updates could be done with standalone SQL, but the migration also updates some property values and data type configuration in ways which would be very unpleasant to implement in pure SQL. In the end you'd still be trying to make the same changes, and if the migration tool is failing then a pure SQL migration would be likely to run into the same errors.

    Could you share your log file from a failed migration to see how far it's getting?

  • Duke 11 posts 102 karma points
    Jan 30, 2020 @ 13:34
    Duke
    0

    Thank you for responding.

    Here is the tail of the log. {"@t":"2020-01-29T18:28:07.1358712Z","@mt":"SQL [{ContextIndex}]: {Sql}","ContextIndex":42055,"Sql":"UPDATE umbracoMediaVersion SET id=55709 WHERE versionId='2c110ce7-f607-4583-af26-b47e94444d01'","SourceContext":"Umbraco.Core.Migrations.Expressions.Execute.Expressions.ExecuteSqlStatementExpression","ProcessId":8244,"ProcessName":"iisexpress","ThreadId":10,"AppDomainId":2,"AppDomainAppId":"LMW3SVC2ROOT","MachineName":"201708-9RQ1JK2","Log4NetLevel":"INFO ","HttpRequestNumber":7,"HttpRequestId":"87e85756-760e-484b-83d0-888b135444ef"} {"@t":"2020-01-29T18:28:07.1388647Z","@mt":"SQL [{ContextIndex}]: {Sql}","ContextIndex":42056,"Sql":"ALTER TABLE [umbracoMediaVersion] ALTER COLUMN [id] INTEGER NOT NULL","SourceContext":"Umbraco.Core.Migrations.Expressions.Execute.Expressions.ExecuteSqlStatementExpression","ProcessId":8244,"ProcessName":"iisexpress","ThreadId":10,"AppDomainId":2,"AppDomainAppId":"LMW3SVC2ROOT","MachineName":"201708-9RQ1JK2","Log4NetLevel":"INFO ","HttpRequestNumber":7,"HttpRequestId":"87e85756-760e-484b-83d0-888b135444ef"} {"@t":"2020-01-29T18:28:15.3707560Z","@mt":"Exception ({InstanceId}).","@l":"Error","@x":"System.Data.SqlClient.SqlException (0x80131904): Cannot insert the value NULL into column 'id', table 'GraphicProducts.com715Upgrade.dbo.umbracoMediaVersion'; column does not allow nulls. UPDATE fails.\r\nThe statement has been terminated.\r\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)\r\n at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)\r\n at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()\r\n at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\\projects\\dotnet\\src\\MiniProfiler.Shared\\Data\\ProfiledDbCommand.cs:line 272\r\n at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.<ExecuteNonQuery>b__31_0() in D:\\a\\1\\s\\src\\Umbraco.Core\\Persistence\\FaultHandling\\RetryDbConnection.cs:line 209\r\n at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.<>c__DisplayClass33_01.

  • Steve Megson 151 posts 1022 karma points MVP c-trib
    Jan 30, 2020 @ 13:58
    Steve Megson
    0

    I think that'll be rows in cmsMedia where the version ID doesn't exist in cmsContentVersion. That's the main thing that'll still kill the migration - orphaned rows that caused no problems in old versions, but stop v8 creating foreign keys and unique indexes.

  • Duke 11 posts 102 karma points
    Jan 30, 2020 @ 14:17
    Duke
    0

    I did find one orphaned record in cmsMedia where the version ID didn't exist in cmsContentVersion. Removing this record then rerunning the migration produced the same result.

    Was removing the record from cmsMedia enough? Where else should I be looking for orphaned records?

    Thank you.

    {"@t":"2020-01-30T14:04:19.5193184Z","@mt":"SQL [{ContextIndex}]: {Sql}","ContextIndex":42056,"Sql":"ALTER TABLE [umbracoMediaVersion] ALTER COLUMN [id] INTEGER NOT NULL","SourceContext":"Umbraco.Core.Migrations.Expressions.Execute.Expressions.ExecuteSqlStatementExpression","ProcessId":8244,"ProcessName":"iisexpress","ThreadId":44,"AppDomainId":3,"AppDomainAppId":"LMW3SVC2ROOT","MachineName":"201708-9RQ1JK2","Log4NetLevel":"INFO ","HttpRequestNumber":8,"HttpRequestId":"74fc72ae-43d4-4dbf-bfa3-7e5792de4d72"}

    {"@t":"2020-01-30T14:04:26.9605194Z","@mt":"Exception ({InstanceId}).","@l":"Error","@x":"System.Data.SqlClient.SqlException (0x80131904): Cannot insert the value NULL into column 'id', table 'dbo.umbracoMediaVersion'; column does not allow nulls. UPDATE fails.\r\nThe statement has been terminated.\r\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)\r\n at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)\r\n at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()\r\n at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\\projects\\dotnet\\src\\MiniProfiler.Shared\\Data\\ProfiledDbCommand.cs:line 272\r\n at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.<ExecuteNonQuery>b__31_0() in D:\\a\\1\\s\\src\\Umbraco.Core\\Persistence\\FaultHandling\\RetryDbConnection.cs:line 209\r\n at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.<>c__DisplayClass33_01.

  • Steve Megson 151 posts 1022 karma points MVP c-trib
    Jan 30, 2020 @ 15:59
    Steve Megson
    0

    Ah, you might get the same error from rows in cmsContentVersion which don't match rows in umbracoNode (then you'd need to check cmsMedia again).

    Depending on which version your site started on, the migration may add 40+ new indexes and foreign keys, so there are a lot of places you could have orphaned rows. Any other errors should be more obvious - it'll tell you that it can't create a foreign key, and it should be obvious where the problem is. This one is unusual because the migration changes version IDs from GUIDs to ints, and the error happens during that change rather than when creating the keys.

    When I get some free time, I'm planning to make a v7 package with health checks to warn about any orphaned rows and other database issues which will stop a migration.

  • Duke 11 posts 102 karma points
    Jan 30, 2020 @ 18:26
    Duke
    0

    Steve, I very much appreciate you taking the time.

    I was able to identify an second orphan media item and remove it from umbracoNode and cmsMedia tables however I'm still not progressing past the same point of failure in the log.

    Am i correct in assuming the migration process is transactional and changes are being rolled back after the failure occurs?

    Jon

  • Steve Megson 151 posts 1022 karma points MVP c-trib
    Jan 31, 2020 @ 10:12
    Steve Megson
    0

    Yes, the migration is done in a transaction and rolls back on failure. It's strange that you're still stuck at the same point, though.

    If you're able to share your database without revealing anything confidential, I'd be happy to take a look. I'm building up quite a collection of stubborn databases that didn't want to migrate.

  • Duke 11 posts 102 karma points
    Feb 12, 2020 @ 19:19
    Duke
    0

    Steve - Thank you for doing what you said you would and writing a tool to check for migration issues. I found and installed the alpha release you published on 2/7/2020. The resulting migration check found the following issues.

    enter image description here

    I have yet to run the fixes but will shortly and will report back.

    Best regards, Jon

  • Duke 11 posts 102 karma points
    Feb 12, 2020 @ 22:45
    Duke
    0

    Steve - the tool did effectively clean up the database however the migration failed do to an different foreign key constraint violation. Here is the tail of the log:

    {"@t":"2020-02-12T22:34:04.4296708Z","@mt":"SQL [{ContextIndex}]: {Sql}","ContextIndex":42446,"Sql":"ALTER TABLE [umbracoContent] ADD CONSTRAINT [FKumbracoContentumbracoNodeid] FOREIGN KEY ([nodeId]) REFERENCES [umbracoNode] ([id])","SourceContext":"Umbraco.Core.Migrations.Expressions.Execute.Expressions.ExecuteSqlStatementExpression","ProcessId":3284,"ProcessName":"iisexpress","ThreadId":12,"AppDomainId":3,"AppDomainAppId":"LMW3SVC2ROOT","MachineName":"201708-9RQ1JK2","Log4NetLevel":"INFO ","HttpRequestNumber":6,"HttpRequestId":"05fd942b-8d48-4f2c-95f6-8f0ec5fcc943"} {"@t":"2020-02-12T22:34:17.2449434Z","@mt":"Exception ({InstanceId}).","@l":"Error","@x":"System.Data.SqlClient.SqlException (0x80131904): The ALTER TABLE statement conflicted with the FOREIGN KEY constraint \"FKumbracoContentumbracoNodeid\". The conflict occurred in database \"GraphicProducts.comLocalDev\", table \"dbo.umbracoNode\", column 'id'.\r\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)\r\n at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)\r\n at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()\r\n at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\\projects\\dotnet\\src\\MiniProfiler.Shared\\Data\\ProfiledDbCommand.cs:line 272\r\n at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.<ExecuteNonQuery>b__31_0() in d:\\a\\1\\s\\src\\Umbraco.Core\\Persistence\\FaultHandling\\RetryDbConnection.cs:line 209\r\n at Umbraco.Core.Persistence.FaultHandling.FaultHandlingDbCommand.<>c__DisplayClass33_01.

  • Steve Megson 151 posts 1022 karma points MVP c-trib
    Feb 14, 2020 @ 15:41
    Steve Megson
    0

    I had only added foreign key checks where I've seen violations in real databases. Out of interest, do you know which version of Umbraco this site started in? I'd expect that foreign key to already exist in your v7 database.

    I've updated the package with more foreign key checks now. Hopefully it's now checking every key that version 8 needs, even if the key should already exist.

  • Brad Wickett 43 posts 124 karma points
    Aug 07, 2020 @ 14:50
    Brad Wickett
    0

    Steve,

    I'm having these exact same issues. I've installed your package but am not able to figure out how to execute the checks. There was no documentation with your package, so after installing in my Umbraco 7.15.5 instance I'm not sure where to execute the checks. When I look at the Developer / Health Checks area there is no new button added for your tool.

    Thank you,

    Brad

Please Sign in or register to post replies

Write your reply to:

Draft