I would think removing (or better: set comment brackets around, in case it crashes the backend and you need to restore) the contents of /config/ExamineIndex.config and /config/ExamineSettings.config should prevent Examine from indexing any content. 

Whether this will speed up you site in any way I'm not sure, I would think the indexing events still occur, but that nothing wil be indexed, as soon as Examine discovers there's no indexes set up.

Regards
Jesper Hauge

@Jesper: Unfortunately this does not completely works. I tried several options to "remove" VirginGames.Web\config\ExamineIndex.config and VirginGames.Web\config\ExamineSettings.config but there are exceptions logged in umbracoLog table.

 

Did you ever find a way to disable examine? We also would like to disable it because we don't need it and rebuilding the indexes also costs performance.

Jeroen

You set the enableDefaultEventHandler = false for your index. This will essentially just leave Examine doing nothing as it won't listen to Umbraco events and thus no indexing.

http://examine.codeplex.com/wikipage?title=UmbracoExamine&referringTitle=Documentation

http://examine.codeplex.com/wikipage?title=Full%20Configuration%20Markup%20%26%20Options&referringTitle=UmbracoExamine

Can I ask why you want to disable it ?

 

@Jeroen, it only rebuilds indexes once when they are not there. Otherwise will just incrementally add to the index when saving/publishing. The performance should be negligeable. 

Hello,

The problem is that we've got a registration form which creates nodes after registration. When a lot of users do this the CPU goes to a 100%. I already started another topic about this (http://our.umbraco.org/forum/developers/api-questions/37022-Publishing-with-API-is-slow), but I recently discoverd that the problem is probably examine which is indexing like crazy. Disabling it probably solves this problem.

Jeroen

Hey Jeroen, that might help the problem with CPU overhead but I suspect the problem is mostly due to the data APIs and their usage. If it's one node created the Examine will only index the one node at a time, not everytime a property is set or anything like that. Let me know how much it does change the performance though as I'd be interested. Examine has also changed the way it works between different versions.

I've set enableDefaultEventHandler = false and removed all the files from \App_Data\TEMP\ExamineIndexes, but the indexes are still being rebuilded. Does it somehow try to rebuild them when it sees there are no files? I've looked through my code and the indexes aren't being rebuilded anywhere. Currently running on Umbraco 4.9. Do you know which version of Examine is with 4.9?

Jeroen

Examine will rebuild the indexes on startup if they don't exist. If you've disabled event handlers they will not be updated with publishing, etc... so they'll just sit there. The rebuilding is a one time thing only since they don't exist during startup. That said, you can disable that too by adding this attribute to your config:

<Examine RebuildOnAppStart="false">

Modify your ExamineIndex.config and ExamineSettings.config file as follows, to disable Lucene indexing entirely. I've done this on Umbraco 6.2.4 If you delete App_Data, the "ExamineIndexes" folder never gets recreated.

config/ExamineIndex.config

<?xml version="1.0"?>
<ExamineLuceneIndexSets>
    <IndexSet SetName="ExternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/External/"/>
</ExamineLuceneIndexSets>

config/ExamineSettings.config

<?xml version="1.0"?>
<Examine RebuildOnAppStart="false">
    <ExamineIndexProviders>
        <providers>
        </providers>
    </ExamineIndexProviders>
    <ExamineSearchProviders defaultProvider="ExternalSearcher">
        <providers>
            <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" enableDefaultEventHandler="false"/>
        </providers>
    </ExamineSearchProviders>
</Examine>

I'll re-iterate again... there shouldn't be any reason to disable Examine and you shouldn't since that is what is used as the media cache. If you don't have up to date indexes your media access will be slow and consume a lot of memory if you have a lot of media in your media tree. Most CPU problems that people blame on Examine/Lucene are actually not Examine's fault.

We have a site that doesn't use Umbraco at all except to log in. I've had to make it an Umbraco site to get that working (though not sure why, I asked questions about that but couldn't get anywhere :( ) - so I don't want the indexer to do anything, ever.

There's simply no point for any overhead at all.

Sounds like a pretty strange case. If Umbraco isn't doing anything (i.e. not saving content or media), then Examine/Lucene won't do anything either, there is no overhead.

I already mentioned mentioned on the previous page:

You set the enableDefaultEventHandler = false for your index. This will essentially just leave Examine doing nothing as it won't listen to Umbraco events and thus no indexing.

It *is* a strange case! That's why it's frustrating that I had to install the whole of Umbraco to get it working...

I've now suddenly started seeing a problem with "OwinStartUpAttribute not specified" even though I've set owin:AutomaticAppStartup to false, though. So I guess I actually *still* don't have a solution to the problem of using just the Umbraco membership provider and none of the rest of Umbraco:(

I'm currently seeing a "device is not ready" error in a shared hosted environment with multiple Umbraco installs. This is apparently related to the Lucene Examine, which is what brought me to this thread.

If my choices are "slow media loading" and "yellow error screen" for my client's websites, I guess I'll have to take the slow media. I'd prefer not to disable Lucene since I'm not sure what else I'll be losing, but my Umbraco sites absolutely must stay up.

Is anyone in this thread familiar with this error? Is there's another way to address this exception without disabling Lucene?

Related: https://our.umbraco.org/forum/umbraco-7/using-umbraco-7/70038-getting-systemio-error-from-lucene-index

https://our.umbraco.org/forum/developers/api-questions/43516-umbraco-on-azure-The-device-is-not-ready-bug

http://issues.umbraco.org/issue/U4-3222

I have started to wonder if this issue on shared hosting is actually to do with some process that runs in the background and touches lucene during some operation potentially corrupting the index. Outside of that I have no idea why I keep getting the "Device Not Ready" error messages at seemingly random intervals

If your host's server setup has your IIS website running off of a network share, then you might have issues with Lucene and IO latency.

In this case you have a couple of options:

In 7.2.8, you can configure your indexers and searchers with the attribute: useTempStorage="LocalOnly"

This needs to be configured for both your indexers and searchers:

For example:

  <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
       supportUnpublished="true"
       supportProtected="true" 
       analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"
       useTempStorage="LocalOnly"/>

  <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
       analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net" 
       useTempStorage="LocalOnly"/>

What this will do is store your indexes on the local hard disk (not the network share) in the ASP.Net temp files location. This also means that anytime you deploy new DLLs or bump your global.asax, the ASP.Net temp files will be removed so when your site restarts it will rebuild the indexes.

If you are on an older version of Umbraco, you can use this project: https://github.com/Shazwazza/UmbracoExamine.TempStorage

which does a very similar thing but requires you update some different configuration options (docs are on that page).

This should resolve IO latency issues for when your hosting provider is hosting your website from a file share (this is true for Azure websites!)

I'd just like to confirm, I have been seeing this error on Azure since May, with a website that has a lot of media in the backoffice, as well as a lot of media on the frontend.

Other symptoms we have seen are that the website freezes (waiting for available socket) after a publish event for up to 60 seconds.

I had originally suspected to be caused by the content editors uploading very large GIF's which ImageResizer was struggle to resize, but disabling cropping on GIF's did not alter performance.

I have updated our Examine configuration to use the recommendations above, we've got monitoring setup, so should be able to see if this resolves the issue, as it's currently failing over quite frequently!

Wishing all a good weekend, Laurence

@Laurence, you haven't provided the actual problem that you are having. What is your Umbraco version? You've written:

I'd just like to confirm, I have been seeing this error on Azure since May

But what is "this error" (there are numerous different errors referenced on this post)

and your umbraco version is ... ? ;)

Please remember, the more information you can provide about your issues the better everyone can help and the faster you can get an answer.

For the device not ready issue for media, what i did was rebuild my index. Click on one of the image then just click save.

Not the best advise but right now i am still developing locally so this is fine.

and yes i got this issue on 7.6.4. and to think this is a new installation.

sorry for late reply to this topic. I have a multiple domain instance of umbraco and when restarting the ISS (or the webserver) I notice some errors connected with Examine Lucene Index.

On some websites (not all of them but just a couple for example), I have an error 500 on the home page and the error is the following: The process cannot access the file 'xxxxxxxxx\App_Data\TEMP\ExamineIndexes\InternalMember\Index\write.lock' because it is being used by another process.

Do you have any suggestion? Which could be the problem?

Can I replace examine with something like elastic search?

Hi Shannon Deminick / Patrick,

We are using umbraco version 6.2.5

Some time we are getting below error ,How to resolve this one?

An unhandled exception occurred

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.IOException: An unexpected network error occurred.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

at System.IO.FileStream.ReadCore(Byte[] buffer, Int32 offset, Int32 count)

at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)

at Lucene.Net.Store.SimpleFSDirectory.SimpleFSIndexInput.ReadInternal(Byte[] b, Int32 offset, Int32 len)

at Lucene.Net.Store.BufferedIndexInput.ReadBytes(Byte[] b, Int32 offset, Int32 len, Boolean useBuffer)

at Lucene.Net.Index.CompoundFileReader.CSIndexInput.ReadInternal(Byte[] b, Int32 offset, Int32 len)

at Lucene.Net.Store.BufferedIndexInput.Refill()

at Lucene.Net.Store.BufferedIndexInput.ReadByte()

at Lucene.Net.Store.IndexInput.ReadVInt()

at Lucene.Net.Index.TermBuffer.Read(IndexInput input, FieldInfos fieldInfos)

at Lucene.Net.Index.SegmentTermEnum.Next()

at Lucene.Net.Index.SegmentTermEnum.ScanTo(Term term)

at Lucene.Net.Index.TermInfosReader.Get(Term term, Boolean useCache)

at Lucene.Net.Index.TermInfosReader.Terms(Term term)

at Lucene.Net.Index.DirectoryReader.MultiTermEnum..ctor(IndexReader topReader, IndexReader[] readers, Int32[] starts, Term t)

at Lucene.Net.Index.DirectoryReader.Terms(Term term)

at Lucene.Net.Search.WildcardTermEnum..ctor(IndexReader reader, Term term)

at Lucene.Net.Search.WildcardQuery.GetEnum(IndexReader reader)

at Lucene.Net.Search.MultiTermQuery.ConstantScoreAutoRewrite.Rewrite(IndexReader reader, MultiTermQuery query)

at Lucene.Net.Search.WildcardQuery.Rewrite(IndexReader reader)

at Lucene.Net.Search.BooleanQuery.Rewrite(IndexReader reader)

at Lucene.Net.Search.BooleanQuery.Rewrite(IndexReader reader)

at Lucene.Net.Search.IndexSearcher.Rewrite(Query original)

at Lucene.Net.Search.Query.Weight(Searcher searcher)

at Lucene.Net.Search.Searcher.Search(Query query, Filter filter, Int32 n, Sort sort)

at Examine.LuceneEngine.SearchResults.DoSearch(Query query, IEnumerable`1 sortField, Int32 maxResults)

at Examine.LuceneEngine.Providers.BaseLuceneSearcher.Search(ISearchCriteria searchParams, Int32 maxResults)

at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMedia(Int32 id)

at Umbraco.Web.PublishedCache.ContextualPublishedCache`1.GetById(Boolean preview, Int32 contentId)

at Umbraco.Web.UmbracoHelper.DocumentById(Int32 id, ContextualPublishedCache cache, Object ifNotFound)

at Umbraco.Web.UmbracoHelper.TypedDocumentById(String id, ContextualPublishedCache cache)

at UmbracoVault.TypeHandlers.MediaTypeHandler.Get(String stringValue)

at UmbracoVault.TypeHandlers.MediaTypeHandler.GetAsType[T](Object input)

--- End of inner exception stack trace ---

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at UmbracoVault.UmbracoWebContext.FillClassProperties[T](T instance, Func`3 getPropertyValue)

at UmbracoVault.uComponents.TypeHandlers.DataTypeGridTypeHandler.GetAsType[T](Object input)

--- End of inner exception stack trace ---

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)

at UmbracoVault.UmbracoWebContext.FillClassProperties[T](T instance, Func`3 getPropertyValue)

at UmbracoVault.UmbracoWebContext.GetItem[T](IPublishedContent n)

at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()

at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)

at UmbracoVault.UmbracoWebContext.GetByDocumentTypeT

at ASP.PageViewsPartialsCommon_BreadCrumbscshtml.Execute() in d:\home\site\wwwroot\Views\Partials\Common_BreadCrumbs.cshtml:line 10

at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()

at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()

at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)

at System.Web.Mvc.Html.PartialExtensions.Partial(HtmlHelper htmlHelper, String partialViewName, Object model, ViewDataDictionary viewData)

at ASP.PageViewsProductDetailcshtml.Execute() in d:\home\site\wwwroot\Views\ProductDetail.cshtml:line 48

at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()

at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()

at System.Web.WebPages.StartPage.ExecutePageHierarchy()

at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)

at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)

at System.Web.Mvc.ControllerActionInvoker.<>cDisplayClass1a.17()

at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func`1 continuation)

at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func`1 continuation)

at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func`1 continuation)

at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>cDisplayClass25.22(IAsyncResult asyncResult)

at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)

at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)

at System.Web.Mvc.Controller.<>cDisplayClass1d.18(IAsyncResult asyncResult)

at System.Web.Mvc.Async.AsyncResultWrapper.<>cDisplayClass4.3(IAsyncResult ar)

at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)

at System.Web.Mvc.Async.AsyncResultWrapper.<>cDisplayClass4.3(IAsyncResult ar)

at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)

at System.Web.Mvc.MvcHandler.<>cDisplayClass8.3(IAsyncResult asyncResult)

at System.Web.Mvc.Async.AsyncResultWrapper.<>cDisplayClass4.3(IAsyncResult ar)

at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)

at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

at System.Web.HttpApplication.<>cDisplayClass285_0.0()

at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)

at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

is working on a reply...