I have a scheduled task set up to run an import on a regular schedule, and this will add and edit any records that are new or changed. Is there a recommended way to use CMSImport Pro to handle record deletions?
My client has a collection of several thousand items (listed in Excel), and I've written a program to generate appropriate "AddAndEdit.xml" and "Subtract.xml" files. I'm using AddAndEdit.xml in conjunction with a scheduled task in CMSImport Pro, and it works well. Before I start custom coding a solution to remove the nodes listed in the Subtract.xml file, is there a built-in way in CMSImport to handle it?
I'm thinking a lot about this feature, but I think it is too much of a risk to build in. Because what happens if you by accident upload an empty file or an old file then all other records will be deleted. Still not sure if this will be a problem for a lot of developers.
Anyway currently it not build in. If you have a seperate xml file containing the key from the datasource I would suggest to take a look at the DataSourceKey column in the CMSImportRelation table. This is a key based on the Datasource Alias(dataadpter you have used in cmsimport, would be XML in your case). , the name of the column containg the key in your datasource containing the value and the actual key value. So based on the key in your substract.xml file you can get the node Id (stored as UmbracoID in the CMSImportRelation table). Then you can delete that document.
Think it's a few lines of code and if you put the substract xml file on a fixed location. Then you can use the Imported Event that is raised when an import is finished to delete these records.
Hope this puts you in the right direction. Please let me know if you have any additional questions
Thanks for that, I can understand why you might be hesitant to include subtraction capabilities. I think, however, that if subtractions required a seperate list of explicit subtractions (ie don't attempt to do any fancy "synchronization") you would be safeguarding against mistaken deletions. I guess it doesn't really matter, as the number of users needing the feature might not be that high.
For what it's worth, I pretty much implemented as you suggested above (working from CMSImportRelation table), but with a couple of gotchas:
1. I subscribe to the ContentImport.Imported event, but this event only fires when there are actually records imported, not when the import process has completed. ie sometimes my add/edit file is actually empty (no records to import tonight!), but I still want to fire the subtraction process at the end of the import task. Any thoughts?
2. I use Lucene indices on my site and learned after trial and error that when deleting a document from the site, I have to follow the following process to ensure it also gets removed from the Lucene index:
Document d = new Document(item); d.UnPublish(); umbraco.library.UnPublishSingleNode(d.Id); d.delete();
(not really CMSImport related)
BTW thanks much for making the CMSImport API so extensible - quality work!
Thanks for your reply, regarding point 1 I think that is related to the import process not being kicked off when the Datasource is empty. Will see if I can fix that in some way.
But your comment regarding your approuch about deleting records via a subtractions file gets me thinking. Maybe I can add an option like the create child definition context menu to create this functionality to achieve this. Will ask around what other users think.Thanks for sharing your ideas.
I thought about unpublishing all records before the import starts, importing and publishing en then removing all unpublished nodes. This would be easiest but seeing i have thousands of records this would be rather slow and inefficient.
Is it possible to index all record id's of the importdata after the import process, comparing them with existing record id's in Umbraco en deleting the difference? (I import and store an original database id and i get all the data every time, not just updates)
Also, do the nodes always get published when update is selected or only if the data is actually updated?
Not yet implemented. It's scheduled for V3. Don't have a release date yet. And there is so much that can go wrong so I want to take the time ;-)
The situation you descibe is possible using the Imported Event you can trigger your own code after the import is finished. You need to know which records didn't exist in the datasource.
And yes pages will be published also when nothing is updated and now I'm thinking about that maybe it would be nice if CMSImport can skip that if the record didn't change .
No it's only an indicator that the import is finished. Easiest might be to loop over all documents since you have key information in there. A bit slow but it's the easiest aprouch.
Document.GetRootDocuments will give you all the root documents then you can loop over the children.
Using CMSImport to delete records
Hi Richard
I have a scheduled task set up to run an import on a regular schedule, and this will add and edit any records that are new or changed. Is there a recommended way to use CMSImport Pro to handle record deletions?
My client has a collection of several thousand items (listed in Excel), and I've written a program to generate appropriate "AddAndEdit.xml" and "Subtract.xml" files. I'm using AddAndEdit.xml in conjunction with a scheduled task in CMSImport Pro, and it works well. Before I start custom coding a solution to remove the nodes listed in the Subtract.xml file, is there a built-in way in CMSImport to handle it?
Thanks
Jonathan
HI Jonathan,
I'm thinking a lot about this feature, but I think it is too much of a risk to build in. Because what happens if you by accident upload an empty file or an old file then all other records will be deleted. Still not sure if this will be a problem for a lot of developers.
Anyway currently it not build in. If you have a seperate xml file containing the key from the datasource I would suggest to take a look at the DataSourceKey column in the CMSImportRelation table. This is a key based on the Datasource Alias(dataadpter you have used in cmsimport, would be XML in your case). , the name of the column containg the key in your datasource containing the value and the actual key value. So based on the key in your substract.xml file you can get the node Id (stored as UmbracoID in the CMSImportRelation table). Then you can delete that document.
Think it's a few lines of code and if you put the substract xml file on a fixed location. Then you can use the Imported Event that is raised when an import is finished to delete these records.
Hope this puts you in the right direction. Please let me know if you have any additional questions
Cheers,
Richard
Hi Richard
Thanks for that, I can understand why you might be hesitant to include subtraction capabilities. I think, however, that if subtractions required a seperate list of explicit subtractions (ie don't attempt to do any fancy "synchronization") you would be safeguarding against mistaken deletions. I guess it doesn't really matter, as the number of users needing the feature might not be that high.
For what it's worth, I pretty much implemented as you suggested above (working from CMSImportRelation table), but with a couple of gotchas:
1. I subscribe to the ContentImport.Imported event, but this event only fires when there are actually records imported, not when the import process has completed. ie sometimes my add/edit file is actually empty (no records to import tonight!), but I still want to fire the subtraction process at the end of the import task. Any thoughts?
2. I use Lucene indices on my site and learned after trial and error that when deleting a document from the site, I have to follow the following process to ensure it also gets removed from the Lucene index:
Document d = new Document(item);
d.UnPublish();
umbraco.library.UnPublishSingleNode(d.Id);
d.delete();
(not really CMSImport related)
BTW thanks much for making the CMSImport API so extensible - quality work!
Jonathan
Hi Jonathan,
Thanks for your reply, regarding point 1 I think that is related to the import process not being kicked off when the Datasource is empty. Will see if I can fix that in some way.
But your comment regarding your approuch about deleting records via a subtractions file gets me thinking. Maybe I can add an option like the create child definition context menu to create this functionality to achieve this. Will ask around what other users think.Thanks for sharing your ideas.
Cheers,
Richard
Hi Richard,
Have you implemented this feature in some way?
I thought about unpublishing all records before the import starts, importing and publishing en then removing all unpublished nodes. This would be easiest but seeing i have thousands of records this would be rather slow and inefficient.
Is it possible to index all record id's of the importdata after the import process, comparing them with existing record id's in Umbraco en deleting the difference? (I import and store an original database id and i get all the data every time, not just updates)
Also, do the nodes always get published when update is selected or only if the data is actually updated?
Harm
HI Harm,
Not yet implemented. It's scheduled for V3. Don't have a release date yet. And there is so much that can go wrong so I want to take the time ;-)
The situation you descibe is possible using the Imported Event you can trigger your own code after the import is finished. You need to know which records didn't exist in the datasource.
And yes pages will be published also when nothing is updated and now I'm thinking about that maybe it would be nice if CMSImport can skip that if the record didn't change .
Best,
Richard
Thanks for the quick response.
Is the external data still accessible in the Imported Event? How can i access it?
Cheers,
Harm
Hi Harm,
No it's only an indicator that the import is finished. Easiest might be to loop over all documents since you have key information in there. A bit slow but it's the easiest aprouch.
Document.GetRootDocuments will give you all the root documents then you can loop over the children.
Cheers,
Richard
is working on a reply...