Just can't get Umbraco to store indexes on local drive
Hi,
I am getting frustrated here because I am trying to get Examine to store it's indexes on the local drive in an Azure Web Site environment and I am getting nowhere. It just refuses to keep the indexes local.
As an example, I have this in my ExamineSettings.config:
Even with the log level set to DEBUG I see no indication that the AzureLocalStorageDirectory is being used. And I see no errors to suggest that I have configured things incorrectly.
Oddly, the Umbraco BackOffice reports that the provider is configured to use the folder I want it to use as in this screenshot:
However, when I check Kudo, there is no such directory:
I am using version 7.4.3. Anyone have any suggestions as to what I might be doing wrong? Could it be permissions, maybe?
Lastly, Kudu doesn't show hidden directories, so if you go back to that temp folder and attempt to do a CD LuceneDir you might see the files are actually there. Regardless, make sure not to use UmbracoExamine.LocalStorage.AzureLocalStorageDirectory which is now obsolete.
Thanks very much for your input and your quick response, I really appreciate it. Just a little while ago I noticed that my server has file handles open to Examine indexes that I can't see from Kudu:
So, I figured that for some reason Kudu was not able to see them. I cannot CD into them as you suggested. But, I guess it doesn't matter as long as they are there and they are being stored locally.
Now, about the AzureLocalStorageDirectory class. Are you saying that the tempStorageDirectory attribute MUST be removed after version 7.4.3? Meaning that it causes or could eventually cause incorrect behaviour?
About the Sync vs LocalOnly. I am a little reluctant to use Sync because I am afraid of the effect on performance of maintaining two indexes. Is this fear unfounded?
And, finally, I am confused about the machine name in the index path. Should the machine name really be left out even in load balanced environments? Intuitively, I would have thought it should be, as you mentioned in your blog, so that Sync will work. But Shannon very recently (in this post) insisted that the machine name is required in load balanced environments (like mine). Surely, that means that it won't sync the index to new instances since the shared indexes won't be found. On the other hand, if we do not use the machine name, how can multiple instances update the same index on the shared drive? Won't there be conflicts?
Thanks again for your clarification. This is really new functionality so documentation is scarce.
Ah, Sorry for the confusion, I didn't realize you were load balancing. For load balancing you will need to follow the documentation, forget everything I said:
Sorry to resurrect an old post, but we're having this exact problem in an Azure load balanced environment on web apps.
We're running 7.4.3 and examine 0.1.69.
We can't get the UmbracoExamine.LocalStorage.AzureLocalStorageDirectory to actually store any indexes locally (presumably this is meant to mean 'on the instance file system - not the vfs').
We do have machinename tokens in our index config.
I'm expecting to see indexes in : d:\local\temp\LuceneDir\[hash]\[our configured path] - is this correct?
There is never a LuceneDir created, and I cannot CD to it in kudu and we're seeing occasional search timeouts that profile traces suggest are speaking to a remote file system.
In the end I stopped using App Services because it wasn't working for me. There is a bug in Windows (https://support.microsoft.com/en-us/kb/3052480) that causes IIS to restart if the App_Data folder is accessed too often - which is exactly what Examine does. (Microsoft says it only happens on Windows Server 2012 R2 but we saw it happening to us in App Services which last I checked was using Windows Server 2012 (not R2).
Anyway, the only way I was able to confirm that Examine was using the local drive was to go into Kudu and check the file handles in use. That told me which files were currently open and I could see clearly that Examine had local indexes. Kudu would not let me CD into the folder, but the file handle list showed they were in use.
Confusingly, the App_Data folder would still show mostly empty folders, as if it was still trying to put indexes there, but full indexes never showed up there.
Thanks very much for the info. Ditching App Services is a pretty big change for us, it feels like swapping one bunch of pain for another slightly different bunch of pain.
We don't have a lot of changes to our indexes, and we think our problems (we do get restarts) are due to vfs/remote storage issues. We're probably going to try LocalCache first and then maybe try swapping out Examine for something else as we don't actually do 'search' but mainly content lookups for a limited and known-in-advance set of ids.
Just can't get Umbraco to store indexes on local drive
Hi,
I am getting frustrated here because I am trying to get Examine to store it's indexes on the local drive in an Azure Web Site environment and I am getting nowhere. It just refuses to keep the indexes local.
As an example, I have this in my ExamineSettings.config:
Even with the log level set to DEBUG I see no indication that the AzureLocalStorageDirectory is being used. And I see no errors to suggest that I have configured things incorrectly.
Oddly, the Umbraco BackOffice reports that the provider is configured to use the folder I want it to use as in this screenshot:
However, when I check Kudo, there is no such directory:
I am using version 7.4.3. Anyone have any suggestions as to what I might be doing wrong? Could it be permissions, maybe?
Thanks for any help you guys can offer.
Regards, Paul.
It's no longer needed to use the
UmbracoExamine.LocalStorage.AzureLocalStorageDirectory- this is all built into the Examine that comes with 7.4.3.For example, one of my indexer configs looks like this now:
<add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine" useTempStorage="LocalOnly"/>You can, of course also use
Syncinstead ofLocalOnly. Also make sure to not have a machine name in youExamineIndex.config. I've blogged about this, and more here: https://cultiv.nl/blog/making-sure-your-umbraco-site-performs-on-azure/Lastly, Kudu doesn't show hidden directories, so if you go back to that temp folder and attempt to do a
CD LuceneDiryou might see the files are actually there. Regardless, make sure not to useUmbracoExamine.LocalStorage.AzureLocalStorageDirectorywhich is now obsolete.Hi Sebastiaan,
Thanks very much for your input and your quick response, I really appreciate it. Just a little while ago I noticed that my server has file handles open to Examine indexes that I can't see from Kudu:
So, I figured that for some reason Kudu was not able to see them. I cannot CD into them as you suggested. But, I guess it doesn't matter as long as they are there and they are being stored locally.
Now, about the AzureLocalStorageDirectory class. Are you saying that the tempStorageDirectory attribute MUST be removed after version 7.4.3? Meaning that it causes or could eventually cause incorrect behaviour?
About the Sync vs LocalOnly. I am a little reluctant to use Sync because I am afraid of the effect on performance of maintaining two indexes. Is this fear unfounded?
And, finally, I am confused about the machine name in the index path. Should the machine name really be left out even in load balanced environments? Intuitively, I would have thought it should be, as you mentioned in your blog, so that Sync will work. But Shannon very recently (in this post) insisted that the machine name is required in load balanced environments (like mine). Surely, that means that it won't sync the index to new instances since the shared indexes won't be found. On the other hand, if we do not use the machine name, how can multiple instances update the same index on the shared drive? Won't there be conflicts?
Thanks again for your clarification. This is really new functionality so documentation is scarce.
Regards, Paul.
Hi Paul,
If you are using Azure Web Apps, then also try to see this documentation
https://our.umbraco.org/documentation/getting-started/Setup/Server-Setup/azure-web-apps
/Dennis
Ah, Sorry for the confusion, I didn't realize you were load balancing. For load balancing you will need to follow the documentation, forget everything I said:
https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/load-balancing/flexible#lucene-examine-configuration
In short: keep what you're using and if you want to use auto-scaling, add the machine name to the config.
Sorry to resurrect an old post, but we're having this exact problem in an Azure load balanced environment on web apps.
We're running 7.4.3 and examine 0.1.69.
We can't get the UmbracoExamine.LocalStorage.AzureLocalStorageDirectory to actually store any indexes locally (presumably this is meant to mean 'on the instance file system - not the vfs').
Our settings; useTempStorage=sync tempStorageDirectory=UmbracoExamine.LocalStorage.AzureLocalStorageDirectory
We do have machinename tokens in our index config.
I'm expecting to see indexes in : d:\local\temp\LuceneDir\[hash]\[our configured path] - is this correct? There is never a LuceneDir created, and I cannot CD to it in kudu and we're seeing occasional search timeouts that profile traces suggest are speaking to a remote file system.
Thanks Brett
Hi Brett.
In the end I stopped using App Services because it wasn't working for me. There is a bug in Windows (https://support.microsoft.com/en-us/kb/3052480) that causes IIS to restart if the App_Data folder is accessed too often - which is exactly what Examine does. (Microsoft says it only happens on Windows Server 2012 R2 but we saw it happening to us in App Services which last I checked was using Windows Server 2012 (not R2).
Anyway, the only way I was able to confirm that Examine was using the local drive was to go into Kudu and check the file handles in use. That told me which files were currently open and I could see clearly that Examine had local indexes. Kudu would not let me CD into the folder, but the file handle list showed they were in use.
Confusingly, the App_Data folder would still show mostly empty folders, as if it was still trying to put indexes there, but full indexes never showed up there.
Regards, Paul.
Hi Paul,
Thanks very much for the info. Ditching App Services is a pretty big change for us, it feels like swapping one bunch of pain for another slightly different bunch of pain.
We don't have a lot of changes to our indexes, and we think our problems (we do get restarts) are due to vfs/remote storage issues. We're probably going to try LocalCache first and then maybe try swapping out Examine for something else as we don't actually do 'search' but mainly content lookups for a limited and known-in-advance set of ids.
Thanks again, Brett
is working on a reply...