flexible load balancing with read only front end servers - Using Umbraco 7

albin 6 posts 27 karma points c-trib

Oct 19, 2015 @ 10:07

Flexible load balancing with read-only front-end servers

I am trying to setup a load balancing environment where the front-end servers have read-only to the database.

Using the guide on https://github.com/umbraco/UmbracoDocs/blob/master/Getting-Started/Setup/Server-Setup/Load-Balancing/flexible-advanced.md I did:

Download and setup a clean umbraco 7.3 (master) with full sql access.
Copied the previouse umbraco folder to a new IIS website (front-end) and changed the connection string to a read-only sql user.
Added MasterServerRegistrar to umbraco master app_code and run SetServerRegistrar on it in ApplicationStarting.
Added FrontEndReadOnlyServerRegistrar to umbraco front-end app_code and run SetServerRegistrar on it in ApplicationStarting.

Current folder structure:

C:\websites\UmbracoCms.7.3.0 master\ (IIS website umb1.local)
                ...
                App_Code\
                    MasterServerRegistrar.cs
                    Startup.cs
                ...
                Web.config - connectionstring contains full sql access


C:\websites\UmbracoCms.7.3.0 front-end\ (IIS website umb2.local)
                ...
                App_Code\
                    FrontEndReadOnlyServerRegistrar.cs
                    Startup.cs
                ...
                Web.config - connectionstring contains read-only sql access

What works:

Master server is always Master.
Front-end server is always Slave.
umbracoServer table is not used anymore.
front-end server only needs read acess to table umbracoNode.

What does not work:

front-end server tries to delete row in umbracoCacheInstruction

    <ExecuteNonQueryWithRetry>b__0 ExecuteAction Execute PruneOldInstructions Sync OnRouteAttempt ProcessRequest
    DECLARE @0 datetime = '2015-10-17T09:49:17';
    DELETE FROM [umbracoCacheInstruction] WHERE utcStamp < @0

front-end server tries to insert row in umbracoCacheInstruction

    <ExecuteScalarWithRetry>b__9 ExecuteAction Insert Insert OnEndRequest <Init>b__9
    DECLARE @0 datetime = '2015-10-19T09:54:31',
            @1 nvarchar(4000) = N'[{"RefreshType":0,"RefresherId":"7b1e683c-5f34-43dd-803d-9699ea1e98ca","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":null,"JsonPayload":null}]',
            @2 nvarchar(4000) = N'DESKTOP-8NBASDQ//LM/W3SVC/3/ROOT [P1788/D2] E6B40EE86R213D60934353451';


INSERT INTO [umbracoCacheInstruction] ([utcStamp], [jsonInstruction], [originated]) VALUES (@0, @1, @2);
SELECT SCOPE_IDENTITY() AS NewID;

Is it possible to run umbraco front-end servers with read-only to the database ?

Copy Link

albin 6 posts 27 karma points c-trib

Oct 19, 2015 @ 13:52

Looks like PruneOldInstructions gets called on every database sync, even if the ServerRole is Slave.

https://github.com/umbraco/Umbraco-CMS/blob/d50e49ad37fd5ca7bad2fd6e8fc994f3408ae70c/src/Umbraco.Core/Sync/DatabaseServerMessenger.cs#L190

Only Master should be clearing old instructions ?

Copy Link

Shannon Deminick 1526 posts 5272 karma points MVP 3x

Oct 20, 2015 @ 16:18

Ahhhh good find. Can you log an issue on the tracker and we'll have it fixed for 7.3.1?

Copy Link

albin 6 posts 27 karma points c-trib

Oct 21, 2015 @ 08:11

Issue created: http://issues.umbraco.org/issue/U4-7283

Copy Link

albin 6 posts 27 karma points c-trib

Oct 21, 2015 @ 09:31

Now to my second question "front-end server tries to insert row in umbracoCacheInstruction".

I have 3 servers (Master, Slave1, Slave2). When I Publish a node on Master a instruction gets inserted to umbracoCacheInstruction. Both Slaves will then try to read the instructions and durning that process a batchmessage gets added to the httpcontext and then read by flushBatch(). This causes the slaves to also write the instructions they read to the database. Conclusion: 3 instructions will be written to umbracoCacheInstruction.

Here is the stack trace of the Slave process when a Master just added a instruction:

ProcessRequest event:

umbraco.dll!Umbraco.Web.BatchedDatabaseServerMessenger.BatchMessage(System.Collections.Generic.IEnumerable<Umbraco.Core.Sync.IServerAddress> servers, umbraco.interfaces.ICacheRefresher refresher, Umbraco.Core.Sync.MessageType messageType, System.Collections.Generic.IEnumerable<object> ids, System.Type idType, string json) Line 133    C#
umbraco.dll!Umbraco.Web.BatchedDatabaseServerMessenger.DeliverRemote(System.Collections.Generic.IEnumerable<Umbraco.Core.Sync.IServerAddress> servers, umbraco.interfaces.ICacheRefresher refresher, Umbraco.Core.Sync.MessageType messageType, System.Collections.Generic.IEnumerable<object> ids, string json) Line 77    C#
Umbraco.Core.dll!Umbraco.Core.Sync.ServerMessengerBase.Deliver(System.Collections.Generic.IEnumerable<Umbraco.Core.Sync.IServerAddress> servers, umbraco.interfaces.ICacheRefresher refresher, Umbraco.Core.Sync.MessageType messageType, System.Collections.Generic.IEnumerable<object> ids, string json) Line 330 C#
Umbraco.Core.dll!Umbraco.Core.Sync.ServerMessengerBase.PerformRefreshAll(System.Collections.Generic.IEnumerable<Umbraco.Core.Sync.IServerAddress> servers, umbraco.interfaces.ICacheRefresher refresher) Line 142   C#
umbraco.dll!Umbraco.Web.Cache.DistributedCache.RefreshAll(System.Guid factoryGuid, bool allServers) Line 208    C#
umbraco.dll!Umbraco.Web.Cache.DistributedCacheExtensions.ClearAllMacroCacheOnCurrentServer(Umbraco.Web.Cache.DistributedCache dc) Line 266  C#
umbraco.dll!Umbraco.Web.Cache.PageCacheRefresher.Refresh(int id) Line 64    C#
Umbraco.Core.dll!Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(System.Guid uniqueIdentifier, string jsonIds) Line 449  C#
Umbraco.Core.dll!Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(System.Collections.Generic.IEnumerable<Newtonsoft.Json.Linq.JToken> jsonArray) Line 408 C#
Umbraco.Core.dll!Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions() Line 255   C#
Umbraco.Core.dll!Umbraco.Core.Sync.DatabaseServerMessenger.Sync() Line 189  C#
umbraco.dll!Umbraco.Web.BatchedDatabaseServerMessenger.UmbracoModule_RouteAttempt(object sender, Umbraco.Web.Routing.RoutableAttemptEventArgs e) Line 54    C#
umbraco.dll!Umbraco.Web.UmbracoModule.OnRouteAttempt(Umbraco.Web.Routing.RoutableAttemptEventArgs args) Line 599    C#
umbraco.dll!Umbraco.Web.UmbracoModule.ProcessRequest(System.Web.HttpContextBase httpContext) Line 117   C#
umbraco.dll!Umbraco.Web.UmbracoModule.Init.AnonymousMethod__13_3(object sender, System.EventArgs e) Line 568    C#

OnEndRequest event:

umbraco.dll!Umbraco.Web.BatchedDatabaseServerMessenger.WriteInstructions(Umbraco.Core.Sync.RefreshInstruction[] instructions) Line 93   C#
umbraco.dll!Umbraco.Web.BatchedDatabaseServerMessenger.FlushBatch() Line 88 C#
umbraco.dll!Umbraco.Web.BatchedDatabaseServerMessenger.UmbracoModule_EndRequest(object sender, System.EventArgs e) Line 66  C#
umbraco.dll!Umbraco.Web.UmbracoModule.OnEndRequest(System.EventArgs args) Line 606  C#
umbraco.dll!Umbraco.Web.UmbracoModule.Init.AnonymousMethod__13_4(object sender, System.EventArgs args) Line 580 C#

I am not sure if this is intentional, but this will prevent Slaves from running with read-only Database Access using the guide "Advanced techniques with Flexible Load Balancing". https://github.com/umbraco/UmbracoDocs/blob/master/Getting-Started/Setup/Server-Setup/Load-Balancing/flexible-advanced.md

Copy Link

Shannon Deminick 1526 posts 5272 karma points MVP 3x

Oct 21, 2015 @ 09:46

It could be another issue, not sure, will need time to investigate. If you could add these details to the issue you've created already that would be great. Then we can have a look at this for the 7.3.1 release. Until then you'll need write access to that table i suppose.

Copy Link

albin 6 posts 27 karma points c-trib

Oct 21, 2015 @ 13:48

I created it as a new issue http://issues.umbraco.org/issue/U4-7285

Copy Link

is working on a reply...

Flag this post as spam?

Flexible load balancing with read-only front-end servers