Bit of background context: site upgraded from v6 > v7.3, started out from a fresh v7 solution and upgraded through backoffice upgrade wizard, all went fine except for some datatypes that we knew would fail (which isn't the issue here)
Currently fighting issue as for each IPublishedContent, we're hitting the database for each .Url property request
I'm not sure why we'd need the database hits, in fact, surprised it does and most probably shouldn't.
Query being executed for each .Url property getter
SELECT umbracoDomains.*, umbracoLanguage.languageISOCode
FROM [umbracoDomains]
LEFT JOIN [umbracoLanguage]
ON [umbracoDomains].[domainDefaultLanguage] = [umbracoLanguage].[id]
WHERE (umbracoDomains.id > 0)
Any ideas where to look for. Probably something is conflicting just not sure where to look for first?
Any ideas where to look/what to hunt for? Configs?
Are you using domain prefixes by any chance? How are the domains set up on the nodes? Could be some sort of conflict that prevents caching the domain query results?
useDomainPrefixes is set to false, as in default 7.3 installation
Domain names are indeed different, but have tried to change those and don't see any changes
RepositoryCacheOptions.GetAllCacheThresholdLimit is set to 100 by default in ctor()
And our solution involves > 100 domains (well, most of them except for 4 to 6 are wildcard'ed), so hence the huge amount. Something I can't change unfortunately...
And because number of domains exceeds the cache threshold limit, it will never get into the cache.
Did a checkout of v7.3 code, set threshold value to 500 and all domains were cached and .Url lookup took 5ms instead of 70ms (!). Makes a huge diff if you're fetching .Url lots of times on a single page.
Oh, btw, same goes for dictionary items, even with a threshold of 500, db is still being hit for each dictionary lookup. Reason: I've got >900 items in dictionary atm.
@Stephan: Any idea how to proceed, any recommendations as how to speed up performance with lots of domains and dictionary items? I'd rather stick to using core as is instead of creating a custom version
Good detective work! I'm trying to think of something that could be done with the current 7.3 version, although ideally what we want is for the cache threshold to be customizable, I guess, for ppl with a large number of domains.
That being said, the larger the number... the more you want to cache them all I guess.
Not sure why the cache threshold would be set to such low numbers. Why not start at something like 100,000? Or, remove the limit entirely?
Making it configurable would be good for extreme use cases, but I'd make it large enough in the first place that it would be very rare that it would ever occur.
linking in this issue as it may well be related - http://issues.umbraco.org/issue/U4-7333 (domain objects in the cache are not being reloaded from database if they are expired/removed)
I think I have traced my issue down to the RepositoryCacheOptions.GetAllCacheValidateCount being set to false so if items get expired from the cache, the code does not reload them from the database when the GetAll function is called.
(for my own education) I don't see how the caching of each item separately give better performance than caching a single enumerable object.
The majority of time the code is going to be searching the collection by domain name rather than a specific id is it not? And if just one cache item is expired then the entire collection needs to be recreated from the database.
Especially as I have one instance of umbraco which has hundreds of domains in it.
I need to talk to Shannon (who implemented that part) but, yes, considering a domain is a very small object in itself, it might make more sense to cache the entire set of domains as one enumerable, whatever the number of domains.
If you have 100 000 domains then... I'm not even sure we want to support that scenario.
Fetching .Url on IPublishedContent hits database
Hi,
Bit of background context: site upgraded from v6 > v7.3, started out from a fresh v7 solution and upgraded through backoffice upgrade wizard, all went fine except for some datatypes that we knew would fail (which isn't the issue here)
Currently fighting issue as for each IPublishedContent, we're hitting the database for each .Url property request
I'm not sure why we'd need the database hits, in fact, surprised it does and most probably shouldn't.
Query being executed for each .Url property getter
SELECT umbracoDomains.*, umbracoLanguage.languageISOCode FROM [umbracoDomains] LEFT JOIN [umbracoLanguage] ON [umbracoDomains].[domainDefaultLanguage] = [umbracoLanguage].[id] WHERE (umbracoDomains.id > 0)
Any ideas where to look for. Probably something is conflicting just not sure where to look for first?
Any ideas where to look/what to hunt for? Configs?
Cheers, Dirk
Uh, this is absolutely wrong, domains should be cached. Is that 7.3.0?
Stephan,
Yes, a 7.3.0 solution
I don't bother downloading source for v7 and see what happens in there, would be great if you know where to look first?
Thanks,
Dirk
Are you using domain prefixes by any chance? How are the domains set up on the nodes? Could be some sort of conflict that prevents caching the domain query results?
Morten,
Also thinking in that direction.
useDomainPrefixes is set to false, as in default 7.3 installation Domain names are indeed different, but have tried to change those and don't see any changes
Current node domain setup example http://www.domain.com/domain
Hold on, maybe I should just strip out http:// (legacy settings)
Let me try that first
[FOLLOW UP] > Doesn't make a difference, tried both http:// and non http:// variant
And up next: Remove /domain from hostname settings
Dirk
Looking into it - weekend response time, though ;-)
Adding some more settings info, cause can't seem to find root cause
Current configuration on v6 live site
And hostnames assigned to top level nodes are always in the form of
(http://)site.com|local/site
Haven't changed any of these settings for v7, did play around with various combo's yet no luck so far :(
I once had a similar with old macroscript being used instead of macropartial. But I guess you have thrown the legacy out.
Aha, found root cause...
RepositoryCacheOptions.GetAllCacheThresholdLimit is set to 100 by default in ctor()
And our solution involves > 100 domains (well, most of them except for 4 to 6 are wildcard'ed), so hence the huge amount. Something I can't change unfortunately...
And because number of domains exceeds the cache threshold limit, it will never get into the cache.
Did a checkout of v7.3 code, set threshold value to 500 and all domains were cached and .Url lookup took 5ms instead of 70ms (!). Makes a huge diff if you're fetching .Url lots of times on a single page.
Oh, btw, same goes for dictionary items, even with a threshold of 500, db is still being hit for each dictionary lookup. Reason: I've got >900 items in dictionary atm.
@Stephan: Any idea how to proceed, any recommendations as how to speed up performance with lots of domains and dictionary items? I'd rather stick to using core as is instead of creating a custom version
Dirk
Good detective work! I'm trying to think of something that could be done with the current 7.3 version, although ideally what we want is for the cache threshold to be customizable, I guess, for ppl with a large number of domains.
That being said, the larger the number... the more you want to cache them all I guess.
Stay tuned.
Not sure why the cache threshold would be set to such low numbers. Why not start at something like 100,000? Or, remove the limit entirely?
Making it configurable would be good for extreme use cases, but I'd make it large enough in the first place that it would be very rare that it would ever occur.
linking in this issue as it may well be related - http://issues.umbraco.org/issue/U4-7333 (domain objects in the cache are not being reloaded from database if they are expired/removed)
Stephen,
I think I have traced my issue down to the RepositoryCacheOptions.GetAllCacheValidateCount being set to false so if items get expired from the cache, the code does not reload them from the database when the GetAll function is called.
(for my own education) I don't see how the caching of each item separately give better performance than caching a single enumerable object.
The majority of time the code is going to be searching the collection by domain name rather than a specific id is it not? And if just one cache item is expired then the entire collection needs to be recreated from the database.
Especially as I have one instance of umbraco which has hundreds of domains in it.
Patrick
(listening)
I need to talk to Shannon (who implemented that part) but, yes, considering a domain is a very small object in itself, it might make more sense to cache the entire set of domains as one enumerable, whatever the number of domains.
If you have 100 000 domains then... I'm not even sure we want to support that scenario.
I'm not familiar with the code, but I'd say a HashSet would probably be more appropriate (i.e., O(1) lookups rather than O(n) lookups).
@Stephen, Have you made any progress on this yet? I was hoping a fix could be in 7.3.2.
is working on a reply...