We have verified that the two servers can communicate by logging into them using the ips above while on an RDP session to the server.
2014-08-22 10:35:37,020 [7] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [Thread 10] An error occurred with the scheduled publishing System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request) at System.Net.WebClient.UploadString(Uri address, String method, String data) at Umbraco.Web.Scheduling.ScheduledPublishing.Start(ApplicationContext appContext) 2014-08-22 10:36:01,876 [7] ERROR Umbraco.Web.Mvc.AdminTokenAuthorizeAttribute - [Thread 28] Failed to format passed in token value System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation. at System.Web.Security.Cryptography.HomogenizingCryptoServiceWrapper.HomogenizeErrors(Func`2 func, Byte[] input) at System.Web.Security.FormsAuthentication.Decrypt(String encryptedTicket) at Umbraco.Core.StringExtensions.DecryptWithMachineKey(String value) at Umbraco.Web.Mvc.AdminTokenAuthorizeAttribute.AuthorizeCore(HttpContextBase httpContext)
Thanks for looking into this for me - FYI the exception dissapears from the logs when I add the new serverName attribute described in the docs. Sorry - I've done this so many times of course I didn't RTFM before doing this!
However the cache isn't updated to the second server.
This is an Azure VMS deployment BTW.
Upon publish I have the following mysterioys message in my logs:
2014-08-23 13:09:40,455 [7] DEBUG Umbraco.Web.BatchedServerMessenger - [Thread 12] Distributed server push completed with no nodes reporting an error
2014-08-23 13:09:44,953 [7] DEBUG Umbraco.Web.Scheduling.Scheduler - [Thread 10] Current server (MHOST-WEB2) detected as a slave, no scheduled processes will execute on this server
I'm testing this locally and unfortunately am having a tough time reproducing.
The serverName attribute is optional - but should be set so that scheduled publishing works properly, if you don't set the attribute then all servers will attempt to perform the scheduled publishing which is not ideal. Also if the serverName or appId attributes are not set then the URL that will be created to perform the scheduled publishing will be determined by:
This is how the code was in previous versions for scheduled tasks, but now we can do this in a better way with the serverName or appId attributes to build the correct URL for the server. The code for this is here:
We've change how distributed calls are done, so they are done in batches instead of sending individual requests to all servers for each action (which based on some things can cause quite a lot of requests to be sent, especially when working with permissions). See task: http://issues.umbraco.org/issue/U4-2633
There are two bits of code that might be the issue. Since you know your slave server's CacheRefresher.asmx is being hit, I wonder if any of this logic is matching and thus exiting?
if (BusinessLogic.User.validateCredentials(login, password) == false)
{
return;
}
//check if this is the same app id as the one passed in, if it is, then we will ignore
// the request - we will have to assume that the cache refeshing has already been applied to the server
// that executed the request.
if (SystemUtilities.GetCurrentTrustLevel() == AspNetHostingPermissionLevel.Unrestricted)
{
//we can only check this in full trust. if it's in medium trust we'll just end up with
// the server refreshing it's cache twice.
if (HttpRuntime.AppDomainAppId == appId)
{
return;
}
}
It would be pretty strange if that was the case though unless you VMs are exact clones of each other and same with your IIS which would mean that you might have the same AppDomainAppId for your IIS sites on each server.
The only error i see is for scheduled publishing, that's because we've fixed that for LB environments, you need to adjust your settings for distributed cache, see updated docs:
I'm back to put the previous version and it works.
I am working with the binaries at the moment. I will try, to download the source code of Umbraco, and perform refresh content after publishing (umbraco.library.RefreshContent() or Services.ContentService.SaveAndPublish()).
When the time comes for publication the node is published, and a distributed call informs the other servers, and the node gets published BUT the cache is not refreshed. This leads to the "Oops: this document is published but is not in the cache (internal error)" - see U4-3145
The node is then in limbo, as it's status is "Published" but it has no URL etc. and isn't in the XML cache
@Genoher the problem that you mentioned is specifically about scheduled publishing. Is that your problem or is it a problem with normal publishing operations?
Also, as I mentioned above you need to adjust your configuration for the new settings that fix scheduled publishing in a load balanced environment:
@Shannon: Sorry for not answering earlier. Finally, we decided to postpone the upgrade.
This week I have taken up this task and at the end I think I found the problem.
First of all, I want to tell you that the problem wasn’t the scheduled publishing because we didn't use it. So the problem was in the cache refresher, which doesn't refresh the cache file "umbraco.config" in the frontends somehow it happen in the backoffice.
In the new version 7.1.6, there is an extra check using the parameter AppId (HttpRuntime.AppDomainAppId).
In our environment, the three instances (backoffice front01, front02) had the same AppDomainAppId.
After changing the Application Id of the different instances the problem was solved.
To have different AppDomainAppId I changed the id of the application (from the advanced configuration of the website).
The code:
//check if this is the same app id as the one passed in, if it is, then we will ignore
// the request - we will have to assume that the cache refeshing has already been applied to the server
// that executed the request.
if (SystemUtilities.GetCurrentTrustLevel() == AspNetHostingPermissionLevel.Unrestricted)
{
//we can only check this in full trust. if it's in medium trust we'll just end up with
// the server refreshing it's cache twice.
if (HttpRuntime.AppDomainAppId == appId)
{
return;
}
}
Yes that is absolutely the issue, we've actually fixed this yesterday. You can also see the latest comments on this thread which also has a workaround posted:
But still we are facing issue of republishing automatically. Just to let you know we are using Citrix load balancer as i read somewhere that sometimes load balancing setup also matters in publish.
Setting up distributedcall for Umbraco load balancing with Umbraco 7
Hello All,
We can't seem to get distributedcalls for load balancing working with v7. Has anyone got this setup and working?
We also notice the exception at the bottom of this post occuring in our logs every few minutes.
Has the mechanism been changed for v7. Is there any additional config required that is different from v6?
Our config looks like this:
We have verified that the two servers can communicate by logging into them using the ips above while on an RDP session to the server.
This is 7.1.5, correct? Have you tried with 7.1.4 before?
For that last error: Does your admin user (id: 0) have a blank value in either the email address, the username or the password by any chance?
Hi Mate - this is 7.1.5 and was a clean install yesterday.
All of those values are set on the user BTW!
And sorry no - have not tried 7.1.4 before, can do if needed.
Alright, not sure about the distributed calls thing, but I've pinged Shannon about this topic.
The docs here apply to v7 by the way, not sure if this would've changed between 7.1.4 and 7.1.5, though this might be something:
http://issues.umbraco.org/issue/U4-581
Seems like the docs have been updated:
https://github.com/umbraco/Umbraco4Docs/commit/40e839f74d20254de0e6c654d60cfd23782bdf2e
With regards to the 404, you testing this on azure websites by any chance? http://issues.umbraco.org/issue/U4-5379
Thanks for looking into this for me - FYI the exception dissapears from the logs when I add the new serverName attribute described in the docs. Sorry - I've done this so many times of course I didn't RTFM before doing this!
However the cache isn't updated to the second server.
This is an Azure VMS deployment BTW.
Upon publish I have the following mysterioys message in my logs:
Over on the target - no error:
2014-08-23 13:09:06,617 [7] DEBUG Umbraco.Web.UmbracoModule - [Thread 12] Begin request: http://mhost-web2.mhost.local/umbraco/Webservices/CacheRefresher.asmx?op=RefreshById.I've RDP'ed onto the first server and hit:
On the second with no issues.
I'm a bit short on time at the mo - but I guess that I need to dig into the code behind CacheRefresher.asmx and possibly add some more logging.
It seems that the request is received by the slave but not processed.
Hi Darren,
I'm testing this locally and unfortunately am having a tough time reproducing.
The serverName attribute is optional - but should be set so that scheduled publishing works properly, if you don't set the attribute then all servers will attempt to perform the scheduled publishing which is not ideal. Also if the serverName or appId attributes are not set then the URL that will be created to perform the scheduled publishing will be determined by:
This is how the code was in previous versions for scheduled tasks, but now we can do this in a better way with the serverName or appId attributes to build the correct URL for the server. The code for this is here:
https://github.com/umbraco/Umbraco-CMS/blob/7.2.0/src/Umbraco.Core/Sync/ServerEnvironmentHelper.cs
So the 404 you were getting was related to this.
We've change how distributed calls are done, so they are done in batches instead of sending individual requests to all servers for each action (which based on some things can cause quite a lot of requests to be sent, especially when working with permissions). See task: http://issues.umbraco.org/issue/U4-2633
So if you are debugging, this is the method that will get called for distributed calls:
umbraco.presentation.webservices.CacheRefresher.BulkRefresh, code is here: https://github.com/umbraco/Umbraco-CMS/blob/7.2.0/src/Umbraco.Web/umbraco.presentation/umbraco/webservices/CacheRefresher.asmx.cs#L27There are two bits of code that might be the issue. Since you know your slave server's CacheRefresher.asmx is being hit, I wonder if any of this logic is matching and thus exiting?
It would be pretty strange if that was the case though unless you VMs are exact clones of each other and same with your IIS which would mean that you might have the same AppDomainAppId for your IIS sites on each server.
Hello Everybody,
I updated from umbraco 7.0.3 to 7.1.6, and the distributed calls doesn't works. The envirodment use certificated SSL.
Servers:
One for backoffice (Umbraco03 - 172.16.101.13), and two balanced frontends (Umbraco01 - 172.16.101.11, Umbraco02 - 172.16.101.12).
When I make a change in a content or create some new content, this is not reflected in the frontends, meanwhile in backoffice it happens.
Configuration:
Errors in logs:
Umbraco03:
Umbraco01:
I'll appreciate any suggestions or test.
PD: Sorry for my english.
The only error i see is for scheduled publishing, that's because we've fixed that for LB environments, you need to adjust your settings for distributed cache, see updated docs:
http://our.umbraco.org/documentation/Installation/load-balancing
specifically:
http://our.umbraco.org/documentation/Installation/load-balancing#Correctconfigforscheduledpublishing&tasks
Though these new settings should not affect your normal distributed caching.
It's showing that it's sending msgs to your servers from 'Umbraco 3':
'Umbraco 1' is reporting that it's receiving the msg:
Thank you very much for the quick response.
I've checked the settings and it seems all right.
I'm back to put the previous version and it works.
I am working with the binaries at the moment. I will try, to download the source code of Umbraco, and perform refresh content after publishing (umbraco.library.RefreshContent() or Services.ContentService.SaveAndPublish()).
I think that I have the same or very similar problem that the next issue. Because the cache is not refreshed. http://issues.umbraco.org/issue/U4-581
Again, thank you very much Shannon ;)
Comment from Dan Booth of issue U4-581
@Genoher the problem that you mentioned is specifically about scheduled publishing. Is that your problem or is it a problem with normal publishing operations?
Also, as I mentioned above you need to adjust your configuration for the new settings that fix scheduled publishing in a load balanced environment:
you need to adjust your settings for distributed cache, see updated docs: http://our.umbraco.org/documentation/Installation/load-balancing
specifically: http://our.umbraco.org/documentation/Installation/load-balancing#Correctconfigforscheduledpublishing&tasks
Hello All,
@Shannon: Sorry for not answering earlier. Finally, we decided to postpone the upgrade. This week I have taken up this task and at the end I think I found the problem.
First of all, I want to tell you that the problem wasn’t the scheduled publishing because we didn't use it. So the problem was in the cache refresher, which doesn't refresh the cache file "umbraco.config" in the frontends somehow it happen in the backoffice.
In the new version 7.1.6, there is an extra check using the parameter AppId (HttpRuntime.AppDomainAppId). In our environment, the three instances (backoffice front01, front02) had the same AppDomainAppId. After changing the Application Id of the different instances the problem was solved.
To have different AppDomainAppId I changed the id of the application (from the advanced configuration of the website).
The code:
Link to commit: https://github.com/umbraco/Umbraco-CMS/commit/3374f9e02301941fe45d14568976adfbf65ff5b0
Hi,
Yes that is absolutely the issue, we've actually fixed this yesterday. You can also see the latest comments on this thread which also has a workaround posted:
http://our.umbraco.org/forum/getting-started/installing-umbraco/56309-Load-balancing-distributed-call-not-updating-cache-(716)
Here's the logged issue:
http://issues.umbraco.org/issue/U4-5491
Here's the fix:
https://github.com/umbraco/Umbraco-CMS/commit/7437240d096dbec6086dea64efeb29b27e19458a
Thanks Genoher for the work-around :-)
Hi Everyone,
Here solution is given specifically to Scheduled publishing. We are facing problem in non scheduled publishing. Everything we have checked based on http://our.umbraco.org/documentation/Installation/load-balancing/
But still we are facing issue of republishing automatically. Just to let you know we are using Citrix load balancer as i read somewhere that sometimes load balancing setup also matters in publish.
Please help on this.
Thanks,
Dhiren
is working on a reply...