I am trying to use CMS Import to transfer content from an existing Umbraco site into a newly created Umbraco site. The old site was running v7.7.8 and the new site runs on v7.15.3. I did wonder if the version might be an issue so I upgraded the older site to also use v7.15.3.
We have about 20,000 articles to import and they are pretty much all just text and images - with a combination of standard text fields, media picker content and macros.
I have the full version of CMS Import and I am able to successfully import the text and other simple fields but I am having trouble with media. I have a property of the doc type which is a media picker. On the old site the format of the images is the node ID but whne I import it the format change to the media URL. I have created 3 different properties which are all of the available media pickers and when I import the same image property into each of these, it is alwasy the URL format, rather than the node ID.
I have also placed the media items in a folder within the app locally so that it can try to import them but it doesn't and times out if I check the 'Show errors' option. I wil rlead through the documentation some more but I'm not sure what I'm doing wrong. Any ideas?
CMSImport can only import Media if it's a url therefore it will convert the media to a Url. What you must do is click the plus sign on the picker and set the media properties there.
I'm not sure what you mean by 'click the plus sign on the picker'.
Do you mean for each individual item that has been imported?
The process has not imported any media items either. I have stored them locally on my new site app and added that folder into CMS Import media settings. The paths look good but if I check 'show errors' in the import mappings setting the import times out.
I should add that the images are usually stored in Azure blob but for the import process I have taken a copy of a sample set and stored locally, as suggested.
I just tried again and have created two extra image properties on my article so that I can see how the data is formatted after import. So now for my imported image, I am mapping to three types of image picker to see what the difference is.
The media picker image data format on the old site (before import) is like this:
<heroImage><![CDATA[39404]]></heroImage>
I checked the + options and specify a node under media to import images ( Import migration). I did this for all of my media picker properties.
The import fails with a message that is has timed out.
If I don't use the extra options the import process completes and the media data looks like this for an imported article:
So, like you said, it has converted the ID to a URL.
If I go to edit the article I get errors as soon as I open the editor:
Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
It appears it cannot find the blob.
And for two of the image properties there is nothing at all displayed (no picker options). For the 'heroTestMulti', which is using the Multiple Media Picker data type, the picker is available but no image has been selected.
None of the images have actually been imported int omy new site. Not sure what I'm doing wrong here.
CMSImport:A record failed to import for aliasheroImage
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteScalar()
at CMSImportLibrary.Helpers.SqlServerHelper.ExecuteScalar[T](String sql, String connectionstring, SqlParameter[] parameters)
at CMSImport.UmbracoDataProvider.Export.Helpers.MediaHelper.GetCroppedUmbracoFileName(String fileName, String connectionString)
at CMSImport.UmbracoDataProvider.Export.Helpers.MediaHelper.GetUmbracoFileName(String fileName, String connectionString)
at CMSImport.UmbracoDataProvider.Content.UmbracoContentDataProvider.ResolveMediaName(String fileName)
at CMSImportLibrary.Helpers.MediaImportHelper.(String , String , AdvancedSettingFieldOptions , Int32 , Boolean , String , Int64 , MediaImportInfo , IMediaResolver )
at CMSImportLibrary.Helpers.MediaImportHelper.(AdvancedSettingFieldOptions , Int32 , String , String , IMediaResolver )
at CMSImportLibrary.Helpers.MediaImportHelper.ImportMedia(Object value, AdvancedSettingFieldOptions advancedFieldOptions, Int32 mediaParentLocation, Boolean allowFolderImport, IMediaResolver mediaResolver)
at CMSImportLibrary.AdvancedSettingControls.Data.MediaPickerSettingsData.Parse(Object value, AdvancedSettingFieldOptions fieldOptions, PropertyInfo propertyInfo)
at CMSImportLibrary.Import.AdvancedSettingsParser.Parse(Object value, PropertyInfo propertyInfo, AdvancedSettingFieldOptions fieldOptions, Dictionary`2 advancedSettings)
at CMSImportLibrary.Providers.ImportProviders.Content.ContentImportProvider.(ImportActions , IContent , Dictionary`2 , AdvancedSettingFieldOptions , PropertyInfo , Object , ImportState , Boolean& )
at CMSImportLibrary.Providers.ImportProviders.Content.ContentImportProvider.(ImportState , Int32 , ImportStatistics , String )
ClientConnectionId:9f875c09-ff60-4457-a63d-321e33b790ec
Error Number:-2,State:0,Class:11
I have upscaled the datbase I am exporting from to Azure S3 because of the timeout issues. I just tried running the import again and 1 out of 3 images has actually been imported! So it seems the import settings must be correct.
It hasn't retained the media section structure though. The image has been placed under 'Image migration' location.
Not sure why only 1 image came through so I'll try again with some new content on the target (old site) database and also without a target node for the media section.
I am now doing a batch of approx 150 items and I am getting quite a lot of images imported! This is a huge leap since last week and I haven't really changed much except scaling up the database and making sure there was no slash before the root folder name in the CMS Import media settings (so just 'CmsImportImage/'). The images are going into my specified 'Import migration' media node but I can move them into the correct folder YYYY/MM structure after each imported batch.
The only problem I have is that he media data format will require some code refactoring to deal with the '/media/**' format.
So, some good progress.
I do have another issue realting to radio button data. When importin is comes through as the text value rather than the datatype INT value. So far I have had to ignore that field as I'm not really sure what to do with it.
I am now getting articles to import with very few errors.
The main issues I have is with radio button data and macro images.
I am looking into the radio button problem but I can't think of any way to successfully import RTE field content (e.g. Body field) with macros containing images which have come from a media picker. I select the 'import images' option when setting up the mappings between RTE fields but when the article is imported, the macro exists but the image is deemed to have been deleted.
Is there a way to increase the timeout on import items?
I'm getting a lot of errors when trying to import images which ARE on disk but for some reason the timeout is regularly breached when trying to import. It could be taking longer because the images go to Aure blob.
Log Error:
CMS Import: A record failed to import for aliasheroImage
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.
Hi Richard, yes I did try that. I can push the timeout limit up a bit more and try again. I'm really not sure why it happens for some images and not others. None of them are particularly large.
That's the only error I have seen when it comes to images not importing.
Is there any caching going on that may make the import process ignore images that have been added to the import media folder more recently?
CMS Import media import problems
Hi,
I am trying to use CMS Import to transfer content from an existing Umbraco site into a newly created Umbraco site. The old site was running v7.7.8 and the new site runs on v7.15.3. I did wonder if the version might be an issue so I upgraded the older site to also use v7.15.3. We have about 20,000 articles to import and they are pretty much all just text and images - with a combination of standard text fields, media picker content and macros.
I have the full version of CMS Import and I am able to successfully import the text and other simple fields but I am having trouble with media. I have a property of the doc type which is a media picker. On the old site the format of the images is the node ID but whne I import it the format change to the media URL. I have created 3 different properties which are all of the available media pickers and when I import the same image property into each of these, it is alwasy the URL format, rather than the node ID.
I have also placed the media items in a folder within the app locally so that it can try to import them but it doesn't and times out if I check the 'Show errors' option. I wil rlead through the documentation some more but I'm not sure what I'm doing wrong. Any ideas?
Hi Chris,
CMSImport can only import Media if it's a url therefore it will convert the media to a Url. What you must do is click the plus sign on the picker and set the media properties there.
Hope this helps,
Richard
Hi Richard,
I'm not sure what you mean by 'click the plus sign on the picker'. Do you mean for each individual item that has been imported?
The process has not imported any media items either. I have stored them locally on my new site app and added that folder into CMS Import media settings. The paths look good but if I check 'show errors' in the import mappings setting the import times out.
Hi Chris,
That config looks good and I think you found the plus sign otherwise you woiuld not have seen the extra options ;-) The path to the media is correct?
Best,
Richard
Hi Richard,
I should add that the images are usually stored in Azure blob but for the import process I have taken a copy of a sample set and stored locally, as suggested.
I just tried again and have created two extra image properties on my article so that I can see how the data is formatted after import. So now for my imported image, I am mapping to three types of image picker to see what the difference is.
The media picker image data format on the old site (before import) is like this:
I checked the + options and specify a node under media to import images ( Import migration). I did this for all of my media picker properties.
The import fails with a message that is has timed out.
If I don't use the extra options the import process completes and the media data looks like this for an imported article:
So, like you said, it has converted the ID to a URL.
If I go to edit the article I get errors as soon as I open the editor:
It appears it cannot find the blob.
And for two of the image properties there is nothing at all displayed (no picker options). For the 'heroTestMulti', which is using the Multiple Media Picker data type, the picker is available but no image has been selected. None of the images have actually been imported int omy new site. Not sure what I'm doing wrong here.
No other errors in the Umbraco Log file during import? i'm just assigning the Stream to the picker so I assume that will work on Azure as well.
Best,
Richard
Hi,
The logs from my most recent effort:
I have upscaled the datbase I am exporting from to Azure S3 because of the timeout issues. I just tried running the import again and 1 out of 3 images has actually been imported! So it seems the import settings must be correct. It hasn't retained the media section structure though. The image has been placed under 'Image migration' location.
Not sure why only 1 image came through so I'll try again with some new content on the target (old site) database and also without a target node for the media section.
Another update...good news.
I am now doing a batch of approx 150 items and I am getting quite a lot of images imported! This is a huge leap since last week and I haven't really changed much except scaling up the database and making sure there was no slash before the root folder name in the CMS Import media settings (so just 'CmsImportImage/'). The images are going into my specified 'Import migration' media node but I can move them into the correct folder YYYY/MM structure after each imported batch.
The only problem I have is that he media data format will require some code refactoring to deal with the '/media/**' format.
So, some good progress.
I do have another issue realting to radio button data. When importin is comes through as the text value rather than the datatype INT value. So far I have had to ignore that field as I'm not really sure what to do with it.
Another update...
I am now getting articles to import with very few errors. The main issues I have is with radio button data and macro images.
I am looking into the radio button problem but I can't think of any way to successfully import RTE field content (e.g. Body field) with macros containing images which have come from a media picker. I select the 'import images' option when setting up the mappings between RTE fields but when the article is imported, the macro exists but the image is deemed to have been deleted.
Is there a way to get this to work?
Is there a way to increase the timeout on import items? I'm getting a lot of errors when trying to import images which ARE on disk but for some reason the timeout is regularly breached when trying to import. It could be taking longer because the images go to Aure blob.
Hi Chris,
This is a SQL Exception. Can't you increase that in web.config?
Best,
Richard
Hi Richard, yes I did try that. I can push the timeout limit up a bit more and try again. I'm really not sure why it happens for some images and not others. None of them are particularly large.
I don't know either but strange that it's throwing a SQL exception Timeout in that case. Shouldn't it be a different error if Azure Times out?
That's the only error I have seen when it comes to images not importing. Is there any caching going on that may make the import process ignore images that have been added to the import media folder more recently?
In case of updates it will only update the file when Bytesize is different from previous imported file.
is working on a reply...