Load balancing distributed call not updating cache (7.1.6)
Hopefully someone can help with this? I am having problems setting up our load balanced environment to work with umbraco (7.1.6). I have followed the documentation but still no luck. My setup is as follows....
1x sql server (shared by both web servers)
2x web servers
Web servers use their own local copy of the code which is synchronised but excludes the App_Data folder (i previously excluded specific files/folders but that didn't work either).
The web servers share an iis config, but have bindings specific for each machine to enable them to communicate internally with each other.
Distributed calls have been setup in the umbracoSettings.config file, this includes setting enable to true, adding the two servers using host names which are added to the hosts file and the IIS bindings, the master server has its server name property set on the server tag in the umbracoSettings.config file.
I am using the umbraco backoffice using server 1 only.
I have monitored each server when i make a change using wireshark and I can see the call come in to server 2 with the correct user/page details which gives a response of 200 ok (with an empty BulkRefreshResponse node) but the cache files/folders are not changed.
The only unusual thing I can see in the web service call is a guidId node which is set to 00000000-0000-0000-0000-000000000000 and an intId node which is set to 0 but I am assuming that the latter is the userId?
Looking at the logs...
There are no errors in the logs and I can see the log item "Submitting calls to distributed servers" on server 1.
There are no items logged on server 2 at the same time as a page is published on server 1.
Server 2 can successfully generate its cache so if i delete the contents of the App_data folder on that server it does regenerate it but it just won't update when a page is published on server 1.
Not sure, I've set this up so many times on my local environment without issue, the problem with LB is that everyone's environment is always different and there are so many moving parts.
First, you should set the serverName on ALL servers listed in your servers config.
Next, you can change your log4net settings to be 'Debug' to output more information, on server 2 with that setting you should at least then see the request coming in for the cacherefresher.asmx. For the time being until you iron out all the issues, might be best to set logging levels to Debug for both of your servers.
Not all of the App_Data folder should be ignored, since you are doing file replication, here's the docs on what to ignore:
We're encountering the same issue and it sounds like we have a very similar setup.
We have confirmed that upon publishing, server 1, calls umbraco/RestServices/ScheduledPublish/Index, then calls umbraco/webservices/cacheRefresher.asmx and upon publishing server 1, the cacheRefresher is called on server 2. The cache files on server 1 are updated correctly, however, the cache files on server 2 (app_data/temp/umbraco.config and app_data/temp/examineindexes/...) are not updated as expected.
Here's our setup
1 SQL Server db instance (SQL Server 2008 R2)
2 web server instances (Windows Server 2008 R2)
Using file replication using DFS to sync the two web instances (umbraco.config is stored in the App_Data/TEMP folder. App_Data/TEMP and App_Data/Logs are the only folders excluded from replication)
Load balancing with instance 1 designated as the back office server - host headers set appropriately
web.config - shared machine key
umbracoSettings.config - distributed calls are enabled, both servers are listed using internal DNS name and server name, i.e.ourServer1.internalip.com
Permissions set correctly on both servers (you can publish the cache fine individually, but distributed publishing is not working)
2014-09-15 11:32:58,949 [17] INFO Umbraco.Core.Publishing.PublishingStrategy - [Thread 30] Content 'Distributed Test' with Id '1623' has been published.
2014-09-15 11:32:59,027 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.UnpublishedPageCacheRefresher on single server instance, message type RefreshById
2014-09-15 11:32:59,214 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.PageCacheRefresher on single server instance, message type RefreshById
2014-09-15 11:32:59,230 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.MacroCacheRefresher on single server instance, message type RefreshAll
2014-09-15 11:32:59,542 [17] INFO Umbraco.Web.BatchedServerMessenger - [Thread 30] Submitting calls to distributed servers
Server 2: IIS Logs 2014-09-15 16:32:59 XXX.107 POST /umbraco/webservices/cacheRefresher.asmx
Umbraco Trace Log No relevant logs - we find this to be very odd.
We're wondering if this error could be due to the Scheduled Publishing & Tasks update made for 7.1.5+ http://our.umbraco.org/documentation/Installation/load-balancing#UmbracoConfiguration - correct config for scheduled publishing & tasks), since it appears that the distributed publishing may now be handled by umbraco/RestServices/ScheduledPublish/Index?
Thank you Shannon. This does sound like the issue we're encountering. We tried to setup the app using the AppDomainAppId method and found out that our AppDomainAppId is the same for both web servers.
is ONLY for scheduled publishing, if you have 2+ sites being load balanced on the same server - which is generally only ever done for testing. You'd 99% of the time, use the serverName attribute.
That said, we do use AppDomainAppId to filter out cache refresher calls so that we don't double cache refresh on the master. All the details are listed on that issue.
Please let me know if the work around works for you (i think it should) and I'll update the LB docs. I'd rather not back port this fix and create new releases just for this one issue when there is a simple work around in place.
Thanks for your help on this one. It looks like this is the reason for our troubles. As we have a shared config the sites HttpRuntime.AppDomainAppId is the same on both servers. Unfortunately this means that your workaround will not work for us but we look forward to the next release which by the look of your fix description should do the trick.
Any ideas when this release will be out? Will it be 7.1.7 or 7.2.0?
hrm, yes with shared config that won't work unless you remove the shared config to get it working - if you need it working now then i guess that's what you'll need to do.
Hello again, ok i have come up with a work around for those using a shared configuration in IIS, unfortunately i think the more servers you have in your cluster the more painful this workaround will be (we only have 2 so not able to test more)....
duplicate your umbraco site in the shared config (you may need a duplicate per server?)
in the original site set the bindings to be specific to the server 1 ip address
in the duplicate site set the bindings to be specific to the server 2 ip address
make sure that each site has its own binding for internal comunication, for me this was just setting up server1.mywebsite.local against the original site and setting up server2.mywebsite.local against the duplicate site these were then added to the hosts file and the umbracoSettings.config file distributedCall section as described in the docs.
This means that when umbraco does its webservice call it will be calling a site with a different appId.
We have several environments, so we didn't feel comfortable with the IIS meta data workaround.
We built a custom version of Umbraco v7.1.6 with this change (only dll changed is umbraco.dll). It's working great for us. We attached it to the issue here http://issues.umbraco.org/issue/U4-5491, so it will hopefully help others.
Load balancing distributed call not updating cache (7.1.6)
Hopefully someone can help with this? I am having problems setting up our load balanced environment to work with umbraco (7.1.6). I have followed the documentation but still no luck. My setup is as follows....
I have monitored each server when i make a change using wireshark and I can see the call come in to server 2 with the correct user/page details which gives a response of 200 ok (with an empty BulkRefreshResponse node) but the cache files/folders are not changed.
The only unusual thing I can see in the web service call is a guidId node which is set to 00000000-0000-0000-0000-000000000000 and an intId node which is set to 0 but I am assuming that the latter is the userId?
Looking at the logs...
Server 2 can successfully generate its cache so if i delete the contents of the App_data folder on that server it does regenerate it but it just won't update when a page is published on server 1.
What have I missed or is this a bug?
Not sure, I've set this up so many times on my local environment without issue, the problem with LB is that everyone's environment is always different and there are so many moving parts.
First, you should set the serverName on ALL servers listed in your servers config.
Next, you can change your log4net settings to be 'Debug' to output more information, on server 2 with that setting you should at least then see the request coming in for the cacherefresher.asmx. For the time being until you iron out all the issues, might be best to set logging levels to Debug for both of your servers.
Not all of the App_Data folder should be ignored, since you are doing file replication, here's the docs on what to ignore:
http://our.umbraco.org/documentation/Installation/load-balancing#Nonreplicatedfiles
We're encountering the same issue and it sounds like we have a very similar setup.
We have confirmed that upon publishing, server 1, calls umbraco/RestServices/ScheduledPublish/Index, then calls umbraco/webservices/cacheRefresher.asmx and upon publishing server 1, the cacheRefresher is called on server 2. The cache files on server 1 are updated correctly, however, the cache files on server 2 (app_data/temp/umbraco.config and app_data/temp/examineindexes/...) are not updated as expected.
Here's our setup
Server 1:
IIS Logs
2014-09-15 16:32:29 XXX.106 POST /umbraco/RestServices/ScheduledPublish/Index - 80 - XXX.106 - 200 0 0 156
2014-09-15 16:32:59 XXX.106 POST /umbraco/webservices/cacheRefresher.asmx - 80 - XXX.106 - 200 0 0 15
Umbraco Trace Logs
2014-09-15 11:32:58,949 [17] INFO Umbraco.Core.Publishing.PublishingStrategy - [Thread 30] Content 'Distributed Test' with Id '1623' has been published.
2014-09-15 11:32:59,027 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Performing distributed call for refresher Umbraco.Web.Cache.UnpublishedPageCacheRefresher, message type: RefreshById, servers: http://[internal1].com/umbraco/webservices/cacheRefresher.asmx;http://[internal2].com/umbraco/webservices/cacheRefresher.asmx, ids: 1623, json:
2014-09-15 11:32:59,027 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.UnpublishedPageCacheRefresher on single server instance, message type RefreshById
2014-09-15 11:32:59,214 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Performing distributed call for refresher Umbraco.Web.Cache.PageCacheRefresher, message type: RefreshById, servers: http://[internal1].com/umbraco/webservices/cacheRefresher.asmx;http://[internal2].com/umbraco/webservices/cacheRefresher.asmx, ids: 1623, json:
2014-09-15 11:32:59,214 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.PageCacheRefresher on single server instance, message type RefreshById
2014-09-15 11:32:59,230 [17] DEBUG Umbraco.Core.Sync.DefaultServerMessenger - [Thread 30] Invoking refresher Umbraco.Web.Cache.MacroCacheRefresher on single server instance, message type RefreshAll
2014-09-15 11:32:59,542 [17] INFO Umbraco.Web.BatchedServerMessenger - [Thread 30] Submitting calls to distributed servers
Server 2:
IIS Logs
2014-09-15 16:32:59 XXX.107 POST /umbraco/webservices/cacheRefresher.asmx
Umbraco Trace Log
No relevant logs - we find this to be very odd.
We're wondering if this error could be due to the Scheduled Publishing & Tasks update made for 7.1.5+ http://our.umbraco.org/documentation/Installation/load-balancing#UmbracoConfiguration - correct config for scheduled publishing & tasks), since it appears that the distributed publishing may now be handled by umbraco/RestServices/ScheduledPublish/Index?
Any advise would be greatly appreciated.
Hi , i think i found the reason for distributed calls not updating the cache in some cases from 7.1.5+, see this issue here with a work around:
http://issues.umbraco.org/issue/U4-5491
Thank you Shannon. This does sound like the issue we're encountering. We tried to setup the app using the AppDomainAppId method and found out that our AppDomainAppId is the same for both web servers.
We'll try this tomorrow and update you.
Thanks
Cool, just to confirm the 'appId' part in the config listed here: http://our.umbraco.org/documentation/Installation/load-balancing#Correctconfigforscheduledpublishing&tasks
is ONLY for scheduled publishing, if you have 2+ sites being load balanced on the same server - which is generally only ever done for testing. You'd 99% of the time, use the serverName attribute.
That said, we do use AppDomainAppId to filter out cache refresher calls so that we don't double cache refresh on the master. All the details are listed on that issue.
Please let me know if the work around works for you (i think it should) and I'll update the LB docs. I'd rather not back port this fix and create new releases just for this one issue when there is a simple work around in place.
Cheers!
Hi Shannon,
Thanks for your help on this one. It looks like this is the reason for our troubles. As we have a shared config the sites HttpRuntime.AppDomainAppId is the same on both servers. Unfortunately this means that your workaround will not work for us but we look forward to the next release which by the look of your fix description should do the trick.
Any ideas when this release will be out? Will it be 7.1.7 or 7.2.0?
Cheers
hrm, yes with shared config that won't work unless you remove the shared config to get it working - if you need it working now then i guess that's what you'll need to do.
7.2 beta should be out in a week or two
Hello again, ok i have come up with a work around for those using a shared configuration in IIS, unfortunately i think the more servers you have in your cluster the more painful this workaround will be (we only have 2 so not able to test more)....
This means that when umbraco does its webservice call it will be calling a site with a different appId.
I hope this helps.
Someone else has posted a workaround here:
http://our.umbraco.org/forum/getting-started/installing-umbraco/55867-Setting-up-distributedcall-for-Umbraco-load-balancing-with-Umbraco-7?p=1#comment193598
might be worth a look.
Thanks Shannon.
None of these will work for us since our environments have the iis meta data shared. This prevents us from setting individual appdomainappid's.
We're going to just create a custom 7.1.6 build incorporating your patch.
Just to double check, this isn't already available, correct?
Yup, see issue: http://issues.umbraco.org/issue/U4-5491
here's the commit: http://github.com/Umbraco/Umbraco-CMS/commit/7437240d096dbec6086dea64efeb29b27e19458a
Thanks for the aditional work around link Shannon. This should work for users who do not have a shared configuration.
@Zac, We too have the iis meta data shared so my work around should work for you but a custom build will certainly work too. :)
Thanks Shannon and Nick.
We have several environments, so we didn't feel comfortable with the IIS meta data workaround.
We built a custom version of Umbraco v7.1.6 with this change (only dll changed is umbraco.dll). It's working great for us. We attached it to the issue here http://issues.umbraco.org/issue/U4-5491, so it will hopefully help others.
is working on a reply...