Load balance site doesn't publish if a server is offline
I'm running load balancing in a test environment before we deploy our site. One of the things I wanted to test is what happens if we pull a server out of the farm and publish content with it online. I expected that it would obviously be out of sync when we brought it back, but I'm suprised that none of the servers update if one is offline.
So in my test environment I have an "admin" install which is used to make content changes, and 2 load balanced servers. I'm able to save and publish content from the admin box and see it change on the load balanced servers as long as they're both running. If I disable IIS on one of them, the content changes only happen on the admin box, but not the 2nd load balanced server.
I'm guessing this has something to do with the order of the servers in my config file. Currently they are:
<server>t1itdexw02</server> (admin server test) <server>t1ittexw02</server> (slave server) <server>172.20.43.21</server> (slave server)
My guess is that publishing dies when it hits the 2nd server and it is unresponsive. It does not continue to the 3rd server which is still online.
I'm digging through the source code to find some answers, but I was wondering if anyone else had seen this behavior.
I think I found the problem. The dispatcher class loops through and refreshes each server. The loop is nested in a try/catch block. If the loop is unsucessful in contacting any one of the servers, then the exception will cause the loop to escape.
public static void Refresh(Guid uniqueIdentifier, int Id) { try { using (CacheRefresher cr = new CacheRefresher()) { foreach (XmlNode n in UmbracoSettings.DistributionServers.SelectNodes("./server")) { cr.Url = "http://" + xmlHelper.GetNodeValue(n) + GlobalSettings.Path + "/webservices/cacheRefresher.asmx"; cr.RefreshById(uniqueIdentifier, Id, _login, _password); } } } catch (Exception ee) { BusinessLogic.Log.Add( BusinessLogic.LogTypes.Error, BusinessLogic.User.GetUser(0), -1, "Error refreshing '" + new Factory().GetNewObject(uniqueIdentifier).Name + "' with id '" + Id.ToString() + "', error: '" + ee.ToString() + "'"); } }
This may be desired behavior, but I think better exception handling would allow the loop to continue, just because one server is unavailable. Maybe calling the try/catch block as a separate method from within the loop?
Load balance site doesn't publish if a server is offline
I'm running load balancing in a test environment before we deploy our site. One of the things I wanted to test is what happens if we pull a server out of the farm and publish content with it online. I expected that it would obviously be out of sync when we brought it back, but I'm suprised that none of the servers update if one is offline.
So in my test environment I have an "admin" install which is used to make content changes, and 2 load balanced servers. I'm able to save and publish content from the admin box and see it change on the load balanced servers as long as they're both running. If I disable IIS on one of them, the content changes only happen on the admin box, but not the 2nd load balanced server.
I'm guessing this has something to do with the order of the servers in my config file. Currently they are:
<server>t1itdexw02</server> (admin server test)
<server>t1ittexw02</server> (slave server)
<server>172.20.43.21</server> (slave server)
My guess is that publishing dies when it hits the 2nd server and it is unresponsive. It does not continue to the 3rd server which is still online.
I'm digging through the source code to find some answers, but I was wondering if anyone else had seen this behavior.
-Ben
I think I found the problem. The dispatcher class loops through and refreshes each server. The loop is nested in a try/catch block. If the loop is unsucessful in contacting any one of the servers, then the exception will cause the loop to escape.
public static void Refresh(Guid uniqueIdentifier, int Id) {
try {
using (CacheRefresher cr = new CacheRefresher())
{
foreach (XmlNode n in UmbracoSettings.DistributionServers.SelectNodes("./server"))
{
cr.Url = "http://" + xmlHelper.GetNodeValue(n) + GlobalSettings.Path + "/webservices/cacheRefresher.asmx";
cr.RefreshById(uniqueIdentifier, Id, _login, _password);
}
}
} catch (Exception ee) {
BusinessLogic.Log.Add(
BusinessLogic.LogTypes.Error,
BusinessLogic.User.GetUser(0),
-1,
"Error refreshing '" + new Factory().GetNewObject(uniqueIdentifier).Name + "' with id '" + Id.ToString() + "', error: '" + ee.ToString() + "'");
}
}
This may be desired behavior, but I think better exception handling would allow the loop to continue, just because one server is unavailable. Maybe calling the try/catch block as a separate method from within the loop?
I'm going to post this as a bug.
Good find and obviously a bug! Thanks for submitting!
is working on a reply...