At two
occasions we have experienced the cmsContentXml table gets wiped. This is resulting
in all the pages being effectively unpublished – the “Link to document” is “#”.
A republish solves the problem, but the website is rather large so republishing
all nodes clocks in about an hour.
We believe
it could be tied to renaming of a DocType, but we are still researching the
issue.
While we
research our self, I’m curious if anyone has similar experiences?
As it happens, I am also researching this issue after it happened to one of our major websites yesterday, although not for the first time. We have seen this happen across several different Umbraco systems, always when "something" has been done in Umbraco. I say "something" because so far there has been no apparent pattern to this.
Currently I am looking at the action of assigning host names to nodes as that was what happened immediately before this happened to us yesterday and we are able to build a test structure which we hope can replicate it. If I do find anything out, then I will gladly let you know because this has been bothering us for some time.
Just to update you, none of our testing so far has produced a definate cause for this problem. In our experience so far there has been no definable set of actions which has led to this, which is to say each time it has happened, it has been APPARENTLY different actions which immediately preceded it. It is of course possible that different actions in the back end of umbraco have similar or the same effects on the database, and this is where our investigation is leading next.
Thanks for the update. We had it happen (on a staging environment) once more. This prompted me to create a work item on Codeplex: http://umbraco.codeplex.com/workitem/30610
We haven’t been able to reproduce the issue in a controlled way.
It happened again in our development environment - again editing doc types. This time I experienced that the save doc type hang. I was a little bit impatient, so I killed the web server. This leads me to believe that the Umbraco code allows for the DB to get in an inconsistent state. Doesn't seem that Umbraco is transacation based...
Thanks for the update. Unfortunately we've had other projects on here, so this isnt a problem I've been able to give much time to recently. The information is really useful though, and I'm hoping to be able to look at it again soon.
I have come up with something of a work-around, though this is certainly not ideal. For other projects, we have a licence of the RedGate SQL Data Compare software. This means I can repopulate the cmsContentXML table from a back-up in around 5 minutes, and then you can simply use the "Republish Entire Site" feature in Umbraco. Then its just a case of publishing the newest nodes.
As I said, this is far from ideal because despite being much quicker than republishing manually, there is obviously a cost involved, plus having a site down for even 5 minutes is not really acceptable.
However I would suggest you look in to the possibility of restoring data from backups using some Data Compare software because this can greatly reduce the effects of this problem, at least until the problem is fixed properly.
Your description of your work-around made me smile - in a damn-that-is-stupid-that-it-is-necessary kind of way. Great use of SQL Data Compare! (I have it on my wishlist.)
It has only happened once in a production environment - we are really careful about editing doc types now - unfortunately we realized what had happened too late to use a backup.
One way to reproduce the issue could be:
Create 2-3000 nodes with the same doc type.
Delete a property on the doc type.
Kill the web server just after save.
But like you, other more pressing issues, are keeping us from investigating this issue thoroughly at the moment.
Glad you liked the work around; I agree that Data Compare is far too big a program for just this problem ("like using a sledgehammer to crack a nut" as we say) but the benefit over directly restoring a backup is that the other tables (and therefore content) in the database can remain unaffected. All my experience of this problem tells me that it is just the cmsContentXml table which is affected, so I am not too concerned over potential integrity issues with it. Plus any problems would be eliminated anyway as pages are updated, saved and published. So far as I can tell, the cmsContentXml table is basically an output, so nothing in the database relies on it for information, its sole job seems to be as a resource to create the XML file on which the front end of the site runs.
The reason I managed to come to this work around was that unfortunately over the last few years, we've had this happen several times on live websites, so a method which would speed up the process was becoming urgent. Like you, the RedGate software has been on my wish list for ages and I finally managed to make a business case for it. That it can be used to manage this problem was part a coincidence and part inspiration!
When I get some time, I will try your suggestion on one of our test systems. From what you're saying, it sounds like the problem may be caused by an interruption in communication with the database. Personally for functions such as we are discussing here, I would have sent all the information to either a temporary table in the database, or directly to a Stored Procedure and then let the database get on with it. In theory then at least even if there was an interruption between the web server and the database, the database would just go ahead and finish what it was doing. I'm guessing Umbraco doesn't work this way.
Whether this is the same bug or not it's a huge worry. Dissappearing sites have affected Umbraco for at least the last three upgrades (search this forum for nonodes.aspx and there's a list of reports over a long period). A CMS that regularly reverts to it's install page can only be considered highly unstable. No one can recommend a CMS that repeatedly crashes whole sites. An admission from the team that this bug exists and is being urgently worked out would be a start.
I have written a post with a workaround fix, see: http://allan-laustsen.blogspot.com/2012/03/umbraco-no-node-exists-cmscontentxml.html Instead of trying to avoid or handle server crashes, database hickups, connection issues, app-pool recycles, etc. I hookup to the document type save event, and then using the tasks system+publishing service to republish the nodes that are used by the document type. In the case any hickups the single node is just published one extra time.
After applying this to our live servers we have had zero problems with out of contentxml-table out of sync ... :-) Hope it helps you too
PS. we are currently running +300 websites and +25000 nodes in a single Umbraco install
#### December 20 - 2012 Update #### Due to the high number of requests I have recived from others also affected by this bug in Umbraco I have supplied release builds (including debug PDB files) of the following umbraco versions 4.7.2 / 4.9.1 / 4.10.1 / 4.11.1 see the post here for the downloads
To my knowledge the issue has not yet been fixed in the umbraco core, and recently I have recived several emails about the issue, from people who need help with this bug, theese requests for help are for both newer and older version of umbraco.
I'm a little hung up at work right now, but I will see if I can have some time before christmas to fix the issue, and hopefully supply it as a package or other form of drop in replacement into the umbraco core, because it's a BIG issue once your install reaches a ceartain size.
If your problem is critical right now, I urge you to try and re-patch the core version 4.9.1 with the patch I tried, however I can not gurantee that it will work. (it requires some testing)
I finally had some time to check up on this bug in the umbarco releases after version 4.7.1.1, and it seems not to have been fixed yet, so I have made release builds, including PDB debug files, of the following umbraco versions 4.7.2 / 4.9.1 / 4.10.1 / 4.11.1 see the post here for the downloads.
Obiwan, is there any chance that you may be able to provide the source for this issue or compile a version of the patch for 4.11.10? We're epxeriencing productions issues right now that are showing very similar symptoms.
Just saw this, which suggests that a fix was introduced as of 4.11.9, so this may not be relevant since we're still seeing similar behavior in 4.11.10.
Reasons for the cmsContentXml table to get wiped?
Hi
At two occasions we have experienced the cmsContentXml table gets wiped. This is resulting in all the pages being effectively unpublished – the “Link to document” is “#”. A republish solves the problem, but the website is rather large so republishing all nodes clocks in about an hour.
We believe it could be tied to renaming of a DocType, but we are still researching the issue.
While we research our self, I’m curious if anyone has similar experiences?
Experienced both on Umbraco 4.7.0 and 4.7.1.
Best regards
Emil
Hi Emil,
As it happens, I am also researching this issue after it happened to one of our major websites yesterday, although not for the first time. We have seen this happen across several different Umbraco systems, always when "something" has been done in Umbraco. I say "something" because so far there has been no apparent pattern to this.
Currently I am looking at the action of assigning host names to nodes as that was what happened immediately before this happened to us yesterday and we are able to build a test structure which we hope can replicate it. If I do find anything out, then I will gladly let you know because this has been bothering us for some time.
Kind Regards,
Rob
Hi Emil,
Just to update you, none of our testing so far has produced a definate cause for this problem. In our experience so far there has been no definable set of actions which has led to this, which is to say each time it has happened, it has been APPARENTLY different actions which immediately preceded it. It is of course possible that different actions in the back end of umbraco have similar or the same effects on the database, and this is where our investigation is leading next.
Cheers,
Rob
Thanks for the update. We had it happen (on a staging environment) once more. This prompted me to create a work item on Codeplex: http://umbraco.codeplex.com/workitem/30610
We haven’t been able to reproduce the issue in a controlled way.
Best regards
Emil
It happened again in our development environment - again editing doc types. This time I experienced that the save doc type hang. I was a little bit impatient, so I killed the web server. This leads me to believe that the Umbraco code allows for the DB to get in an inconsistent state. Doesn't seem that Umbraco is transacation based...
Best regards
Emil
Hi Emil,
Thanks for the update. Unfortunately we've had other projects on here, so this isnt a problem I've been able to give much time to recently. The information is really useful though, and I'm hoping to be able to look at it again soon.
I have come up with something of a work-around, though this is certainly not ideal. For other projects, we have a licence of the RedGate SQL Data Compare software. This means I can repopulate the cmsContentXML table from a back-up in around 5 minutes, and then you can simply use the "Republish Entire Site" feature in Umbraco. Then its just a case of publishing the newest nodes.
As I said, this is far from ideal because despite being much quicker than republishing manually, there is obviously a cost involved, plus having a site down for even 5 minutes is not really acceptable.
However I would suggest you look in to the possibility of restoring data from backups using some Data Compare software because this can greatly reduce the effects of this problem, at least until the problem is fixed properly.
Cheers,
Rob
Hi Rob
Your description of your work-around made me smile - in a damn-that-is-stupid-that-it-is-necessary kind of way. Great use of SQL Data Compare! (I have it on my wishlist.)
It has only happened once in a production environment - we are really careful about editing doc types now - unfortunately we realized what had happened too late to use a backup.
One way to reproduce the issue could be:
But like you, other more pressing issues, are keeping us from investigating this issue thoroughly at the moment.
Best regards
Emil
Hi Emil
Glad you liked the work around; I agree that Data Compare is far too big a program for just this problem ("like using a sledgehammer to crack a nut" as we say) but the benefit over directly restoring a backup is that the other tables (and therefore content) in the database can remain unaffected. All my experience of this problem tells me that it is just the cmsContentXml table which is affected, so I am not too concerned over potential integrity issues with it. Plus any problems would be eliminated anyway as pages are updated, saved and published. So far as I can tell, the cmsContentXml table is basically an output, so nothing in the database relies on it for information, its sole job seems to be as a resource to create the XML file on which the front end of the site runs.
The reason I managed to come to this work around was that unfortunately over the last few years, we've had this happen several times on live websites, so a method which would speed up the process was becoming urgent. Like you, the RedGate software has been on my wish list for ages and I finally managed to make a business case for it. That it can be used to manage this problem was part a coincidence and part inspiration!
When I get some time, I will try your suggestion on one of our test systems. From what you're saying, it sounds like the problem may be caused by an interruption in communication with the database. Personally for functions such as we are discussing here, I would have sent all the information to either a temporary table in the database, or directly to a Stored Procedure and then let the database get on with it. In theory then at least even if there was an interruption between the web server and the database, the database would just go ahead and finish what it was doing. I'm guessing Umbraco doesn't work this way.
Anyway, good luck with this!
Cheers,
Rob
Sounds like this bug: http://our.umbraco.org/forum/jobs/jobs/26147-Website-redirecting-to-noNodesaspx detailed here and many other places: http://blog.darren-ferguson.com/2010/11/2/working-around-the-%27no-nodes%27-issue-with-umbraco-45x where noNodes.aspx reappears, caused perhaps by a DB connection error, and is 'fixed' by a republish. One workaround involves replacing noNodes.aspx with a code snippet that republishes the site. It might not be suitable for a large site that takes much time to republish.
Whether this is the same bug or not it's a huge worry. Dissappearing sites have affected Umbraco for at least the last three upgrades (search this forum for nonodes.aspx and there's a list of reports over a long period). A CMS that regularly reverts to it's install page can only be considered highly unstable. No one can recommend a CMS that repeatedly crashes whole sites. An admission from the team that this bug exists and is being urgently worked out would be a start.
Hi Jed
Nope. Not the same bug. I have been bitten by the other one as well. Here the error is a 404 page, and not the noNodes.aspx.
Thanks for chipping in on this!
Best regards
Emil
Hi
I have written a post with a workaround fix, see: http://allan-laustsen.blogspot.com/2012/03/umbraco-no-node-exists-cmscontentxml.html
Instead of trying to avoid or handle server crashes, database hickups, connection issues, app-pool recycles, etc. I hookup to the document type save event, and then using the tasks system+publishing service to republish the nodes that are used by the document type. In the case any hickups the single node is just published one extra time.
After applying this to our live servers we have had zero problems with out of contentxml-table out of sync ... :-)
Hope it helps you too
PS. we are currently running +300 websites and +25000 nodes in a single Umbraco install
#### December 20 - 2012 Update ####
Due to the high number of requests I have recived from others also affected by this bug in Umbraco I have supplied release builds (including debug PDB files) of the following umbraco versions 4.7.2 / 4.9.1 / 4.10.1 / 4.11.1 see the post here for the downloads
Fantastic work!
Hi, has this issue been fixed? I'm experiencing it in 4.9.1
The cmsContentXml table is not getting wiped out though that I am noticing, just all the entries for images .
I'm not positive that it is coinciding with docType edits.
Obiwan, I saw your blog post. I'm not sure how to install the patch.
Can you help me?
Hi #jacob phillips
To my knowledge the issue has not yet been fixed in the umbraco core, and recently I have recived several emails about the issue, from people who need help with this bug, theese requests for help are for both newer and older version of umbraco.
I'm a little hung up at work right now, but I will see if I can have some time before christmas to fix the issue, and hopefully supply it as a package or other form of drop in replacement into the umbraco core, because it's a BIG issue once your install reaches a ceartain size.
If your problem is critical right now, I urge you to try and re-patch the core version 4.9.1 with the patch I tried, however I can not gurantee that it will work. (it requires some testing)
/Allan Laustsen
Hi all
I finally had some time to check up on this bug in the umbarco releases after version 4.7.1.1, and it seems not to have been fixed yet, so I have made release builds, including PDB debug files, of the following umbraco versions 4.7.2 / 4.9.1 / 4.10.1 / 4.11.1 see the post here for the downloads.
/Allan Laustsen
Thanks for your help on this. So far so good.
Obiwan, is there any chance that you may be able to provide the source for this issue or compile a version of the patch for 4.11.10? We're epxeriencing productions issues right now that are showing very similar symptoms.
Best,
Nik
Just saw this, which suggests that a fix was introduced as of 4.11.9, so this may not be relevant since we're still seeing similar behavior in 4.11.10.
http://issues.umbraco.org/issue/U4-493
Thanks!
is working on a reply...