We have a website with an authoring server (umbraco 7.3.2) and a web node for the public website (ie. No umbraco backoffice). The method of pushing content changes from authoring to the public website is a hangover from umb4, but ultimately it achieved by copying the umbraco.config file out to the public website. The public website is also configured to use the authorinng umbraco database (so that it starts up successfully).
This has an intermittent issue where the public website will start throwing the following exception:
DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (20159: "[{"RefreshType":3,"RefresherId":"55698352-dfc5-4dbe-96bd-a4a0f6f77145","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[5636]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[5636]","JsonPayload":null}]"). Instruction is being skipped/ignored
System.ArgumentException: Invalid or unsupported character in number
at SupportClass.Number.ToInt64(String s)
at Lucene.Net.Index.SegmentInfos.GetCurrentSegmentGeneration(String[] files)
at Lucene.Net.Index.SegmentInfos.GetCurrentSegmentGeneration(Directory directory)
at Lucene.Net.Index.IndexReader.IndexExists(Directory directory)
at Examine.LuceneEngine.Providers.LuceneIndexer.AddNodesToIndex(IEnumerable1 nodes, String type) in x:\Projects\Examine\Examine\Projects\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 890
at Examine.ExamineManager._ReIndexNode(XElement node, String type, IEnumerable1 providers) in x:\Projects\Examine\Examine\Projects\Examine\ExamineManager.cs:line 200
at Umbraco.Core.Cache.CacheRefresherBase`1.Refresh(Int32 id)
at Umbraco.Web.Cache.UnpublishedPageCacheRefresher.Refresh(Int32 id)
The only way to resolve it is to remove the App_Data folder so that Examine indexes can be rebuilt. Is this a known issue? thanks
You probably have a field set to numeric or something and passing in a non numeric value. Best I can do is google for you, the issue is coming from Lucene:
"Invalid or unsupported character in number" in "SupportClass.Number.ToInt64"
hi Shannon - thanks for the info. If that was the case I would expect a cold boot of the website to cause the same exception (i.e. a full rebuild of examine indexes). However, that is not the case - a cold boot is successful. The problem im seeing happens at sporadic times, possibly after the scheduled recycle of the application pool that is built into IIS. It seems that something is corrupting the examine index content and the first exception I see is
Source value:
Umbraco.Web.PublishedCache.XmlPublishedCache.XmlPublishedContent ---> System.ArgumentException: Invalid or unsupported character in number: -
at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem(String cacheKey, Func1 getCacheItem, Nullable1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, CacheDependency dependency)
at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem(String cacheKey, Func1 getCacheItem, Nullable1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles)
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetCacheValues(Int32 id, Func2 func)
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMedia(Int32 id)
at Umbraco.Web.PublishedCache.ContextualPublishedCache1.GetById(Boolean preview, Int32 contentId)
at Umbraco.Web.PublishedCache.ContextualPublishedCache.GetById(Int32 contentId)
at System.Linq.Enumerable.WhereSelectArrayIterator2.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator1.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
BTW - the website now uses the 'flexible load balancing' architecture available in 7 as per the documentation. One instance is the Master, and the other is the Slave (public facing website, umbraco backoffice not available here). Content synchronisation seems to work correctly generally speaking, and then at a random time (could be days or up to a week plus) the examine index will corrupt on the Slave instance as per the exceptions above.
If that was the case I would expect a cold boot of the website to cause the same exception (i.e. a full rebuild of examine indexes)
Maybe not, a full rebuild might treat data slightly differently than a single re-indexing of a node.
The error with the text: "Invalid or unsupported character in number" is definitley a Lucene error that is bubbling up. I would suggest that you look through your logs for when this type of thing happens to see if you can find the document that is causing problems. You might need to change the Examine level debug output in log4net to Debug.
Do you have any custom analyzers or anything like that?
hi Shannon, no there are no custom analyzers or anything, its just what umbraco ships with out of the box.
I debugged this issue on my local dev using the problematic index files and the lucene 2.9.4 source, and its the filename of one of the segments files it's complaining about, not actually content in the index.
Trying to determine what umbraco Examine was trying to do that resulted in the corrupt index is proving difficult even with an elevated logging level.
2016-04-12 10:23:48,141 [R#:(null)] [35] ERROR DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (22392: "[{"RefreshType":3,"RefresherId":"55698352-dfc5-4dbe-96bd-a4a0f6f77145","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[6842]","JsonPayload":null}]"). Instruction is being skipped/ignored
System.ArgumentException: Invalid or unsupported character in number: -
at SupportClass.Number.ToInt64(String s)
at Lucene.Net.Index.SegmentInfos.GetCurrentSegmentGeneration(String[] files)
at Lucene.Net.Index.SegmentInfos.GetCurrentSegmentGeneration(Directory directory)
at Lucene.Net.Index.IndexReader.IndexExists(Directory directory)
at Examine.LuceneEngine.Providers.LuceneIndexer.IndexAll(String type) in x:\Projects\Examine\Examine\Projects\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 812
at UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild()
at Examine.LuceneEngine.Providers.LuceneIndexer.AddNodesToIndex(IEnumerable1 nodes, String type) in x:\Projects\Examine\Examine\Projects\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 893
at Examine.ExamineManager._ReIndexNode(XElement node, String type, IEnumerable1 providers) in x:\Projects\Examine\Examine\Projects\Examine\ExamineManager.cs:line 200
at Umbraco.Core.Cache.CacheRefresherBase1.Refresh(Int32 id)
at Umbraco.Web.Cache.UnpublishedPageCacheRefresher.Refresh(Int32 id)
at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds)
at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable1 jsonArray)
at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions()
as some background to this project, the codebase was originally an umbraco 4 site which is now 7.3.8. At the time this ticket was written the website still used some bespoke code (developed when the website was based on umbraco 4) to automate the distribution of umbraco content (read: umbraco.config file) from the authoring site to the public website.
This has since been removed from the project as this is now functionality built into 7 i.e. flexible load balancing.
Please ignore the original comment:
The method of pushing content changes from authoring to the public website is a hangover from umb4, but ultimately it achieved by copying the umbraco.config file out to the public website.
This is an old topic but for the benefit of anyone searching for information on the error "DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions...", I have had machines stuck in this state three times now, logging that error every 2 minutes, and I've just worked out how to get past it.
The solution that worked for me was quite simple. New instances did not report the error, and when they started up they reported "no last synced id found", so I went and found the last synced id for the machine that was reporting errors and deleted it. It's a file on disk at ~/App_Data/TEMP/DistCache/{machinename}/*-lastsynced.txt. I then restarted the w3wp.exe process to force a cold boot, and it came back working just like the new instances do.
umbraco - authoring server issue
Hi
We have a website with an authoring server (umbraco 7.3.2) and a web node for the public website (ie. No umbraco backoffice). The method of pushing content changes from authoring to the public website is a hangover from umb4, but ultimately it achieved by copying the umbraco.config file out to the public website. The public website is also configured to use the authorinng umbraco database (so that it starts up successfully).
This has an intermittent issue where the public website will start throwing the following exception:
The only way to resolve it is to remove the App_Data folder so that Examine indexes can be rebuilt. Is this a known issue? thanks
You probably have a field set to numeric or something and passing in a non numeric value. Best I can do is google for you, the issue is coming from Lucene:
"Invalid or unsupported character in number" in "SupportClass.Number.ToInt64"
You can see in their code what that is doing and where it is called: https://svn.apache.org/repos/asf/lucene/lucene.net/trunk/C%23/src/Lucene.Net/SupportClass.cs
hi Shannon - thanks for the info. If that was the case I would expect a cold boot of the website to cause the same exception (i.e. a full rebuild of examine indexes). However, that is not the case - a cold boot is successful. The problem im seeing happens at sporadic times, possibly after the scheduled recycle of the application pool that is built into IIS. It seems that something is corrupting the examine index content and the first exception I see is
hi again Shannon, This website is on 7.3.4 and I'm trying to determine if the following tickets may resolve this issue?
7.3.5 http://issues.umbraco.org/issue/U4-7643
7.3.6 http://issues.umbraco.org/issue/U4-7682
7.3.7 http://issues.umbraco.org/issue/U4-7857 http://issues.umbraco.org/issue/U4-7869
BTW - the website now uses the 'flexible load balancing' architecture available in 7 as per the documentation. One instance is the Master, and the other is the Slave (public facing website, umbraco backoffice not available here). Content synchronisation seems to work correctly generally speaking, and then at a random time (could be days or up to a week plus) the examine index will corrupt on the Slave instance as per the exceptions above.
Hi, just replying inline:
Maybe not, a full rebuild might treat data slightly differently than a single re-indexing of a node.
The error with the text: "Invalid or unsupported character in number" is definitley a Lucene error that is bubbling up. I would suggest that you look through your logs for when this type of thing happens to see if you can find the document that is causing problems. You might need to change the Examine level debug output in log4net to Debug.
Do you have any custom analyzers or anything like that?
hi Shannon, no there are no custom analyzers or anything, its just what umbraco ships with out of the box.
I debugged this issue on my local dev using the problematic index files and the lucene 2.9.4 source, and its the filename of one of the segments files it's complaining about, not actually content in the index.
Trying to determine what umbraco Examine was trying to do that resulted in the corrupt index is proving difficult even with an elevated logging level.
fuller stack trace FYI
HI Shannon,
as some background to this project, the codebase was originally an umbraco 4 site which is now 7.3.8. At the time this ticket was written the website still used some bespoke code (developed when the website was based on umbraco 4) to automate the distribution of umbraco content (read: umbraco.config file) from the authoring site to the public website.
This has since been removed from the project as this is now functionality built into 7 i.e. flexible load balancing.
Please ignore the original comment:
That is no longer the case for this project.
Thanks
This is an old topic but for the benefit of anyone searching for information on the error "DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions...", I have had machines stuck in this state three times now, logging that error every 2 minutes, and I've just worked out how to get past it.
The solution that worked for me was quite simple. New instances did not report the error, and when they started up they reported "no last synced id found", so I went and found the last synced id for the machine that was reporting errors and deleted it. It's a file on disk at ~/App_Data/TEMP/DistCache/{machinename}/*-lastsynced.txt. I then restarted the w3wp.exe process to force a cold boot, and it came back working just like the new instances do.
is working on a reply...