Hmm suddenly I'm getting the error in the logs:

Provider=InternalIndexer, NodeId=-1
System.Exception: Cannot index queue items, the index is currently locked,, IndexSet: InternalIndexSet

Also somehow the Examine Management dashboard won't load anymore. I get this exception:

Failed to retrieve indexer details

The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.

EXCEPTION DETAILS:

System.InvalidOperationException: The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.

I used to work. Didn't really change anything.

Jeroen

The reason I got these lock exceptions was because of my local storage. I didn't configure it so multiple indexes used the same folder. Here is my AzureDirectoryFactory in which this is fixed:

public class AzureDirectoryFactory : IDirectoryFactory
{
    public Directory CreateDirectory(LuceneIndexer indexer, string luceneIndexFolder)
    {
        return new AzureDirectory(
            CloudStorageAccount.Parse(ConfigSettings.AzureConnectionString), 
            ConfigSettings.AzureUmbracoExamineContainerName,
            GetLocalStorageDirectory(luceneIndexFolder),
            rootFolder: indexer.IndexSetName);
    }

    private static Directory GetLocalStorageDirectory(string luceneIndexFolder)
    {
        var catalogDir = new DirectoryInfo(luceneIndexFolder);
        if (!catalogDir.Exists)
        {
            catalogDir.Create();
        }

        return FSDirectory.Open(catalogDir);
    }
}

Jeroen

The above code worked local, but not on a load balance setup in Azure. The following code does work and should be the final version:

public class AzureDirectoryFactory : IDirectoryFactory
{
    public Directory CreateDirectory(LuceneIndexer indexer, string luceneIndexFolder)
    {
        return new AzureDirectory(
            CloudStorageAccount.Parse(ConfigSettings.AzureConnectionString), 
            ConfigSettings.AzureUmbracoExamineContainerName,
            GetLocalStorageDirectory(indexer.IndexSetName),
            rootFolder: indexer.IndexSetName);
    }

    private static Directory GetLocalStorageDirectory(string name)
    {
        var appDomainHash = ToMd5(HttpRuntime.AppDomainAppId);
        var cachePath = Path.Combine(
            Environment.ExpandEnvironmentVariables("%temp%"), 
            "LuceneDir",
            appDomainHash, 
            "App_Data", 
            "TEMP", 
            "ExamineIndexes", 
            name);
        var azureDir = new DirectoryInfo(cachePath);
        if (azureDir.Exists == false)
        {
            azureDir.Create();
        }

        return new SimpleFSDirectory(azureDir);
    }

    private static string ToMd5(string stringToConvert)
    {
        // create an instance of the MD5CryptoServiceProvider
        var md5Provider = new MD5CryptoServiceProvider();

        // convert our string into byte array
        var byteArray = Encoding.UTF8.GetBytes(stringToConvert);

        // get the hashed values created by our MD5CryptoServiceProvider
        var hashedByteArray = md5Provider.ComputeHash(byteArray);

        // create a StringBuilder object
        var stringBuilder = new StringBuilder();

        // loop to each each byte
        foreach (var b in hashedByteArray)
        {
            // append it to our StringBuilder
            stringBuilder.Append(b.ToString("x2").ToLower());
        }

        // return the hashed value
        return stringBuilder.ToString();
    }
}

Jeroen

On the master server I get the follow error occasionally:

System.Exception: Cannot index queue items, the index is currently locked,, IndexSet: InternalIndexSet

Not sure if this is a big problem. The indexes still seem to be up to date. The error happens here in the code: https://github.com/Shazwazza/Examine/blob/master/Projects/Examine/LuceneEngine/Providers/LuceneIndexer.cs#L1452

Somehow in the logs it happens every day for all indexes at 05:21.

Jeroen

This error also happens frequently after updating our QA through CI:

System.ApplicationException: Could not create an index searcher with the supplied lucene directory ---> System.IO.FileNotFoundException: segments_3 ---> Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   at Examine.Directory.AzureDirectory.AzureDirectory.OpenInput(String name) in d:\w\3\4d52359c6a32a67a\Sources\Project.Web.Core\ExamineAzure\AzureDirectory.cs:line 253
   --- End of inner exception stack trace ---
   at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
   at Lucene.Net.Index.DirectoryReader.Open(Directory directory, IndexDeletionPolicy deletionPolicy, IndexCommit commit, Boolean readOnly, Int32 termInfosIndexDivisor)
   at UmbracoExamine.UmbracoExamineSearcher.OpenNewReader()
   at Examine.LuceneEngine.Providers.LuceneSearcher.ValidateSearcher(Boolean forceReopen) in X:\Projects\Examine\Examine\Projects\Examine\LuceneEngine\Providers\LuceneSearcher.cs:line 288
   --- End of inner exception stack trace ---
   at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem(String cacheKey, Func`1 getCacheItem, Nullable`1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, CacheDependency dependency)
   at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem(String cacheKey, Func`1 getCacheItem, Nullable`1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles)
   at Umbraco.Core.Cache.DeepCloneRuntimeCacheProvider.GetCacheItem(String cacheKey, Func`1 getCacheItem, Nullable`1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles)
   at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetCacheValues(Int32 id, Func`2 func)
   at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMedia(Int32 id)
   at Umbraco.Web.PublishedCache.ContextualPublishedCache`1.GetById(Boolean preview, Int32 contentId)
   at Project.Web.Core.PropertyConverters.MultipleMediaPickerConverter.ConvertSourceToObject(PublishedPropertyType propertyType, Object source, Boolean preview) in d:\w\3\4d52359c6a32a67a\Sources\Project.Web.Core\PropertyConverters\MultipleMediaPickerConverter.cs:line 80
   at Umbraco.Web.PublishedCache.XmlPublishedCache.XmlPublishedProperty.<.ctor>b__1()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Umbraco.Web.PublishedPropertyExtension.GetValue[T](IPublishedProperty property, Boolean withDefaultValue, T defaultValue)
   at Project.Models.ContentModels.Home.get_HeaderImage() in d:\w\3\4d52359c6a32a67a\Sources\Project.Models\ContentModels\Home.generated.cs:line 66
   at Project.Web.Core.Mappings.TypeConverters.HomePage.HomePageToHeroMoleculeConvertor.Convert(ResolutionContext context) in d:\w\3\4d52359c6a32a67a\Sources\Project.Web.Core\Mappings\TypeConverters\HomePage\HomePageToHeroMoleculeConvertor.cs:line 67
   at AutoMapper.MappingExpression`2.<>c__DisplayClass15.<ConvertUsing>b__14(ResolutionContext context)
   at AutoMapper.Mappers.TypeMapMapper.Map(ResolutionContext context, IMappingEngineRunner mapper)
   at AutoMapper.MappingEngine.AutoMapper.IMappingEngineRunner.Map(ResolutionContext context)
   --- End of inner exception stack trace ---
   at AutoMapper.MappingEngine.AutoMapper.IMappingEngineRunner.Map(ResolutionContext context)
   at AutoMapper.MappingEngine.Map[TDestination](Object source, Action`1 opts)
   at Project.Web.Core.Controllers.HomeController.Hero() in d:\w\3\4d52359c6a32a67a\Sources\Project.Web.Core\Controllers\HomeController.cs:line 122
   at lambda_method(Closure , ControllerBase , Object[] )

After clearing the Examine indexes in blob and restarting the website it works again.

Jeroen

I've got it working well too and I can optimize my InternalIndexer but I only have 20 documents so I will try with a much bigger site and see how it works. This is really great stuff though!

Jeroen, what query were you doing for your bench marking and how many nodes, could you share the code for that?

I have been focused on ensuring that the cold boot 0 documents in index issue is a thing of the past with Azure Directory as that's the issue that has plagued me I hadn't looked at performance.

It looks like your query is taking 3 seconds apposed to 200 milliseconds which is a lot and doesn't make much sense as the indexes are on disk the second time around.

I'll fire this into a project with thousands of documents and try it with your test code and let you know how it performs.

I tried switching the local storage to RAMDirectory, but the performance is still very slow. So it's not the location of the local files.

public Directory CreateDirectory(LuceneIndexer indexer, string luceneIndexFolder)
{
    return new AzureDirectory(
        CloudStorageAccount.Parse(ConfigSettings.AzureConnectionString), 
        ConfigSettings.AzureUmbracoExamineContainerName,
        new RAMDirectory(),
        rootFolder: indexer.IndexSetName);
}

Jeroen

Thanks for the update! I'm on holidays for a couple of weeks after this week so I wont have any time to investigate this before then. I might suggest either attaching break points in the code to discover where the bottleneck is or add some profiling logging. Perhaps the issue is that the logic is somehow reversed and it's not reading correctly from only the local files, or while reading from the local files it is still requesting an unecessary status from Blob storage.

Hi Shannon and Jeavon,

Did any of you tried Azure Directory? Did you also have performance issues?

Jeroen

Sorry :( I really haven't had any time to try to diagnose, i really want to but seem to have too many other urgent tasks to complete/fix.

Hi Shannon,

I've just send you an email with a complete solution with test Azure blob storage to reproduce it. Thought I would also post parts of the email here so other people can also try it.

I've created a branch of the 1-1 multilingual example which now also has a search function: https://github.com/jbreuer/1-1-multilingual-example/tree/AzureDirectory

Steps to reproduce:

1. Unzip the attachment
2. Open \Sources\24days\24days.sln and build the solution
3. Configure the blob storage in the web.config
4. Run the project from VS
5. Login into the Umbraco backoffice: localhost:55988/umbraco/. Username: admin - Password: testtest
6. Go to the Examine dashboard and rebuild the indexes. This should create the blob storage container for you and the local index files.
7. Go to the search page with trace enabled: http://localhost:55988/nl/zoeken/?q=hoofdkwartier&umbDebugShowTrace=true
8. You can see that the search action takes about 200ms. If you remove the directoryFactory from the index providers in ExamineSettings.config and rebuild the examine indexes you can see it will than take about 10 ms.
9. All AzureDirectory classes are in \Sources\Umbraco.Extensions\ExamineAzure if you want to debug anything.

Search in my solution is slow with almost no content. So it's probably not related to the amount of content. Jeavon said he didn't have performance issues so maybe it's something I configured wrong in my solution.

Jeroen

I've contacted Richard Astbury who is maintaining the AzureDirectory project on Github. He also downloaded 1-1 multilingual example branch with AzureDirectory and is able to reproduce the issue.

It's still unclear how to fix the performance issues, but here are some interesting findings (parts of a Skype chat):

• What it's doing is a HEAD request for each blob, to see if it's changed. It detects that the files haven't changed, and then loads them out of the cache. My lucene knowledge isn't that great (I didn't write this library!) but I think that some files are immutable. So perhaps we could always go to the cache for some files. We could also set a cache expiry for the mutable ones. Apparently the segment files are immutable. I thought there was a file which lists all the segments, but I'm not sure. Maybe it just scans the directory (container).
• When I run the unit tests it doesn't make the HEAD requests to storage for every file. So my guess is that this is a lucene configuration thing.
• Ah - I think I've found something interesting. If you hold on to an instance of IndexSearcher, the cache seems to work correctly. If you create a new IndexSearcher for every search, it does the head requests. But it seems that IndexSearcher is a singleton.

So we still haven't found a solution. Shannon maybe the above findings are helpful for you?

Jeroen

Besides the performance issues it seems that the Examine and AzureDirectory combination is pretty stable. I only get this error in my local log every once in a while:

2016-10-03 15:35:54,690 [P8280/D2/T7] ERROR UmbracoExamine.DataServices.UmbracoLogService - Provider=InternalIndexer, NodeId=-1
System.Exception: Cannot index queue items, the index is currently locked,, IndexSet: InternalIndexSet

Not sure if that's a problem.

Jeroen

Most of the calls to blob storage seem to be triggered by this line in examine:

https://github.com/Shazwazza/Examine/blob/b5a17b34981ee41067da287232eb6a646bae3193/Projects/Examine/LuceneEngine/LuceneExtensions.cs#L29

    [SecuritySafeCritical]
    public static ReaderStatus GetReaderStatus(this IndexReader reader)
    {
        ReaderStatus status = ReaderStatus.NotCurrent;
        try
        {
            status = reader.IsCurrent() ? ReaderStatus.Current : ReaderStatus.NotCurrent; // <--- this line
        }
        catch (AlreadyClosedException)
        {
            status = ReaderStatus.Closed;
        }
        return status;
    }

I'm wondering (I don't know) if this should be on a background thread, and called periodically?

I have tried changing this method, so it always returns ReaderStatus.Current which reduces the number of calls.

However, one call remains, which is to list the blobs (AzureDirectory.List()).

Which is called (indirectly) by IndexReader.IndexExists() here:

https://github.com/Shazwazza/Examine/blob/b5a17b34981ee41067da287232eb6a646bae3193/Projects/Examine/LuceneEngine/Providers/LuceneSearcher.cs#L277

    [SecuritySafeCritical]
    private bool ValidateSearcher(bool forceReopen)
    {
        if (!IndexReader.IndexExists(GetLuceneDirectory())) return false; // <-- this line

        if (!forceReopen)
        {
            if (_reader == null)
            ...

I wonder whether this result could be cached, as it's an unlikely scenario that an index is deleted while the site is running (again, an assumption).

Commenting out this line stops AzureDirectory from making any calls to blob storage for a search.

I hope this helps. I'm afraid I'm not an expert on Lucene!

@Richard as there are dual writes to the cache directory and the Azure Directory would it not make sense to change the check on the reader status to look at the cache directory - which presumably is quicker?

I'm just guessing :)

A publish would happen on the "master" CMS server and then typically each "slave" would receive a notification to update it's cache and indexes.

I need to look at the code a bit further but I assume it is fine that the check happens against the cache directory of the Azure reader somehow.

I need a break for a bit first though - this is quite full on :)

I'm kind of blocked as the fork of AzureDirectory by Richard here:

https://github.com/azure-contrib/AzureDirectory

uses Lucene 3 whereas Examine uses 2.9.4.1

I guess that the project here https://azuredirectory.codeplex.com/ is still Lucene 2.9.4.1 but doesn't contain some of the other fixes and improvements that Richard has made.

My rough idea would be to make Examine aware of the Azure Directory object and do any checks on the reader status against the cache directory.

I tried just commenting/removing the offending code yesterday and it does take queries down from 300ms on a small collection of content to 1ms, so I think it is fair to say that this is the bottleneck.

But I could do with some input from Shannon because I think making Examine aware of the AzureDirectory kind of defeats the purpose of supporting the factory pattern in the first place.

In the short term, one could just override the default Lucene Searcher and do some trickery over there. I'm not sure how much benefit has been added in the latest versions of Lucene and AzureDirectory (that doesn't ship with Umbraco) in the course of this thread.

Hi Darren,

Did you check the AzureDirectory version that is in Examine? https://github.com/Shazwazza/Examine/tree/master/Examine.Directory.AzureDirectory

The commit description is: Adds AzureDirectory backported to work with Lucene 2.9 with the sync changes made and a MutexManagers that is per-directory instance.

It's also used here: https://github.com/jbreuer/1-1-multilingual-example/tree/AzureDirectory/Sources/Umbraco.Extensions/ExamineAzure

Jeroen

I blogged about how you can have Examine indexes in Azure Blob Storage. It also has a working example: http://24days.in/umbraco-cms/2016/umbraco-edge-case-stories/#examine

It still has the same performance problems.

Jeroen

Hello,

Does anyone have a status update for this thread? Has someone succeeded in improving the performance when using AzureDirectory?

Jeroen

Hi all,

I've been working on the Examine.Directory.Sync project - which is very similar to the AzureDirectory stuff and have discovered much of the same things (and much more) as @Richard.

I've also logged this issue on the AzureDirectory project: https://github.com/azure-contrib/AzureDirectory/issues/23 based on some "interesting" lucene discoveries. Lucene is fairly odd the way that it works in many cases so getting things right isn't always how it seems.

As @Richard mentions, there are many ways in which Directory.ListAll() or Directory.List() is executed. Everytime this happens it lists all files in the folder. When it comes to Blob storage, this would of course start getting very slow. This is executed anytime a call to:

• IndexReader.IndexExists() is called
• IndexReader.IsCurrent() is called

Both of these methods are executed very often which is probably a very large factor of why it is slow. Examine calls these methods when:

• IndexReader.IndexExists() whenever a search is performed and whenever an index writer is created (once per index per app)
• IndexReader.IsCurrent() is called whenever a search is performed to check if the current index searcher stale so that it can be re-opened if there's been new index operations done

Until now, I naively thought that the performance of IndexExists() and IsCurrent() would be very fast since many Lucene engines call these quite a bit for all sorts of reasons, as it turns out however this is not true. I would suspect that Lucene 4.x (which is coming out later this year) probably has dealt with these things in a better way. In any case, I can look into updating Examine to deal with this in a much better way.

Firstly, I will change all calls to IndexReader.IndexExists() in the index searcher, to only check this once if it actually does exist. Then I need to change how the reader is refreshed. In the 4.x version of Lucene.Net a SearchManager/NrtManager does all of this for us but since we're stuck with Lucene.Net 2.9 and 3.x we don't have this luxury like Java users do. In Examine 2.0-beta these classes have been back ported but I'm still not confident in them and I'm not sure if they'll survive that release. In the latest commits of Examine 0.80.0-beta we already have a large performance improvement where a Commit will only happen on a sliding time schedule, see: https://github.com/Shazwazza/Examine/issues/62 I can use this same code to have a fast sliding time schedule to check if the reader needs to be re-opened. This is basically what Lucene's SearchManager does! In 0.80.0 we also utilize NRT readers which are much faster to open.

So I think getting the last things done in 0.80.0 will make this work well, stay tuned and I'll let you know when it's ready.

@jeroen,

I haven't had a chance to test your solution but I've done a ton of work to (hopefully) make this run fast with blob storage. I've pushed some Nuget packages for you to test, install (or update) in this order:

Install-Package Examine -Pre -Version 0.1.80-beta02
Install-Package Examine.Directory.Sync -Pre -Version 1.0.0-beta02
Install-Package Examine.Directory.AzureDirectory -Pre -Version 1.0.0-beta02

I've updated the zure directory stuff to use the latest WindowsAzure.Storage package too.

I've changed quite a lot regarding reading files and IO processing. I'll also make another PR to the original AzureDirectory repo if all of this works as planned. I've tested the SyncDirectory which is essentially the same thing but instead of storing the master index in blob, it just stores the master index in a folder (i.e. on Azure websites it would be on a remote file share)

Please let me know how it goes if you get a chance to test, otherwise i'll see if i can find time tomorrow.

I couldn't wait so I tested this immediately :-).

I've upgraded the 1-1 multilingual example AzureDirectory branch to the latest NuGet packages: https://github.com/jbreuer/1-1-multilingual-example/commits/AzureDirectory

After this I did all the steps to reproduce the performance problem. Now performance is perfect! Sometimes a little spike to 20ms, but most of the time it's so fast it doesn't even show up on the trace log.

Thank you for your hard work Shannon. I'm really happy you found some time to help us.

When I try to reindex the content I sometimes get the following error:

Cannot delete C:\Users\jnb\AppData\Local\Temp\LuceneDir\005b309ffbac5a50adddf56d6cd63a9b\App_Data\TEMP\ExamineIndexes\ExternalIndexSet\_0.cfs

So I probably need to tweak my AzureDirectoryFactory. Maybe it's also possible to ship that with the Examine.Directory.AzureDirectory package?

Jeroen

Hi Shannon,

Yes the 'cannot delete' error only happens with the debugger attached. Good to know this is normal.

Great that the AzureDirectoryFactory will be part of the code base. That means we probably don't need to code anything to use AzureDirectory. Only some config changes.

I think merging Examine.Directory.Sync into the Examine core is a good idea if there are no additional dependencies.

Let me know if I can help testing with the issues that are left.

Jeroen

I see Examine 0.1.80-beta03 and Examine.Directory.AzureDirectory 1.0.0-beta03 are out on NuGet. Will do some tests this weekend :-).

Jeroen

I've upgraded the packages to beta3. I tried to uninstall Examine.Directory.Sync because it's in the core, but I couldn't because Examine.Directory.AzureDirectory 1.0.0-beta03 still depends on it.

I changed the directory factory to the one shipped with Examine.Directory.AzureDirectory: https://github.com/jbreuer/1-1-multilingual-example/commit/5a15cc9c2ef035e64763a854277aeb511f897346#diff-2047c3df736265061843bec54bd24783

But that currently throws the following exception:

Could not load type 'Examine.LuceneEngine.Providers.IDirectoryFactory' from assembly 'Examine, Version=0.1.80.0, Culture=neutral, PublicKeyToken=null'.

So I'm not sure if there is something wrong with my configuration or that some of the packages still need to be update.

Jeroen

@Shannon we would also like to use azure for Examine to improve the performance and cpu usage when we swap the slots after deploying to web apps.

Hi Shannon do you have a status update on the new AzureDirectory package? I've got the feeling that it's really close to being finished. Would really like to start using it.

Jeroen

Hello,

I don't think a version will be released where we can store Examine indexes in blob storage using AzureDirectory.

I think at CodeGarden 17 a nice alternative was shown for Examine on web apps in Azure. It needed some changes to the Examine config files. Anyone got an example?

Jeroen

An update about Examine with Azure Directory (Blob Storage) has been posted here: https://our.umbraco.com/forum/developers/api-questions/56303-Status-of-Examine-and-Azure-blob-storage-providers#comment-294969

Jeroen

Finally a non-alpha version of Blob Storage based Lucene indexes for Examine has been released. More info: https://shazwazza.com/post/examine-and-azure-blob-storage/

Jeroen

is working on a reply...