Yes, using Luke the query takes between 300 and 2000us (so, nothing alarming here)

What do you mean by the following?

So think you are saying that it's not the index query that is causing the delay...

Rendering the results is not the problem. Debugging my code, and stepping through everything, timing everything, I see that it is in fact the search() method from the BaceLuceneSearcher class that is taking its time. When search() finally sends me a response, my code executes fairly quick afterwords (not the bottleneck).

May I point your attention for these lines in the first post:

Pastebin for Lucene query with a single searchword: http://pastebin.com/5PFqkDzC

Pastebin for Lucene query with multiple searchword: http://pastebin.com/CHFMsnvk

I will try to create a simple working code example that replicates my issue.

Wow, I got majorly sidetracked by other projects, but now I'm back.

Looking at the thread, I really don't know what you wan't the razor code, the view file has nothing to do with this. As stated previously, it is the BaseLuceneSearcher in Examine that is taking its time.

How do I know that this is where the problem lies? Well, when running the code in debug mode, stepping through every step in the search everything runs smoothly until I reach the Search() method in the BaseLuceneSearcher class. When I step over this part of the code, it takes several seconds for the debugger to get any further. Here I also see a big difference in searching with one word, or with serveral.

Furthermore, I have found some other and even more frustrating examples that leads me think that something is entirely off with the BaseLuceneSearcher class.

I was optimizing my queries, removing a lot of *-symbols, and removing search for words that didn't provide any useful data. I was shocked to find that in Luke the query went from 180ms to 100ms, but when feeding it to Examine it takes 6 seonds to execute for each. Here I had expected it to take 6 seconds for the non-optimized query, and maybe 4 for the optimized. But no, Examine doesn't care, it takes 6 seconds for each query.

Old query: http://pastebin.com/bcjK2FHi

Optimized query: http://pastebin.com/2bhhK75K

(I know that the queries aren't pretty, I'll try to make them easier to look at when I get the time)

FYI: I tried to create a code-example, but I couldn't really figure out how to create the test in a way that it would be reproduce able for you guys. If you have any ideas for how to create such a test, please do let me know, and I will throw some time at it.

Thanks for you thorough response Shannon. I will make a proper response later today.

For now I will give you a quick response to some of the things you are mentioning.

In your notes above, how are you timing this 6 seconds?

Examine it takes 6 seonds to execute for each

You keep mentioning it's when you are debugging/breakpointing. You cannot effectively measure any timing while you are debugging/breakpointing, Visual Studio will be busy doing a zillion things during this time like loading in symbols, etc...

I'm aware of that, and I might not have been explaining that clearly enough, that is my fault.

What I noticed that was, that in the compiled product, it took me <20ms to get a http-response when searching for one word, and >5s to get a http-response with several words. This I see by opening Developer Tools in chrome, and looking at the "Network" tab. (I can go more into depth with this if needed)

I then turned to my code, debugging it so that I could step through every method to see what it was that was taking so long. Here I found that the Search() method in the BaseLuceneSearcher was taking it's time to respond. I know that it takes a lot of time when debugging, so I have tested both debugging when searching for one word, and when searching for multiple, and when searching for multiple words it takes considerably longer to execute the Search() method then when I step through it, searching for only one word.

Next part is what I will take my time to write a proper response to:

More importantly is to know how you are:

• Creating and executing this query
• What are you trying to achieve with these multi-word queries?

The response will come later today.

EDIT: Regarding your link, it seem that the forum doesn't like "_" in the URLS, I guess you ment to refer to this link: Apache Lucene - Query Parser Syntax

More importantly is to know how you are:

• Creating and executing this query
• What are you trying to achieve with these multi-word queries?

Lets start this one backwords.

• What are you trying to achieve with these multi-word queries?

The end goal is that when a user searches for the query "flemming andreasen", I want to look for content that matches any of these requirements:

1. "flemming andreasen" as a direct hit
2. flemming as a direct hit
3. andreasen as a direct hit
4. any term that begins with flemming
5. any term that begins with andreasen

In the end-result I also want to weight results differently so that results are prioritized according to the above list

Next part:

... how you are creating and executing this query

I'm not sure how thorough I should be here, I will try to make it a mix of abstract and concrete, and then hope that you guys will ask me to clarify if something is not clear enough.

Generating the query

When something is typed into the search-bar it is send to the searchController via JSON. Here the string is split up into different keywords (and an failed attempt for search-strings). When this is done queries are created, this is where the sub-query is created.

To generate the query I have written my own library which simply put lets me do things like:

new LuceneQuery(allowLeadingWildcards: true, occurance: BooleanClause.Occur.SHOULD, useStopWords: false).GroupedOr(new[] { "fullName" }, terms)

This piece of pseudoCode would then create the previously discussed sub-query:

(((fullName:flemming andreasen* fullName:"flemming andreasen" fullName:"flemming andreasen" fullName:flemming andreasen~0.7 fullName:flemming* fullName:andreasen* fullName:flemming fullName:andreasen fullName:flemming fullName:andreasen))^5.0)

Searching

When the final query is created it is then handed to Examines BaseLuceneSearcher Search() method like this:

var searcher = (MultiIndexSearcher)ExamineManager.Instance.SearchProviderCollection["CombinedIndexSearcher"];
var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.And);
...
var results = searcher.Search(searchCriteria.RawQuery(rawQuery))

Showing results

When the results are ready they are returned to the view as a JSON object like this:

return Json(
            new
            {
                success = true,
                results = results
            }, JsonRequestBehavior.AllowGet
        );

Here I have some Angular code ready to receive and display it

Is this something that you can use?

Hi

I have been working on shaving off the duplicates and trying to make the query do what I wanted it to. I ended up with the following query:

(title:"flemming andreasen") (((title:"flemming andreasen"~0.7))^0.4) (((title:*flemming*))^0.9) (((title:andreasen*))^0.9) (((title:flemming title:andreasen))^0.2)

Still no change, the query still took several seconds du execute (it did get a little faster, it got down to 4-5 seconds in execute time, but nothing like what I had hoped). Then I looked more closely at the "Apache Lucene - Query Parser Syntax" and saw that I couldn't do "fuzzy search" on a sentence, but instead could do a "proximity search", which seems to have the requirement of being at least of one or greater...

You cant do this:

title:"flemming andreasen"~0.7

but you can do this:

title:"flemming andreasen"~1

Removing my attempt to do fuzzy search on a sentence made a BIG difference. Now I'm down to 600-800ms.

(title:"flemming andreasen") (((title:*flemming*))^0.9) (((title:andreasen*))^0.9) (((title:flemming title:andreasen))^0.2)

This is puzzling me, could this part: title:"flemming andreasen"~0.7 really be doing that much harm?

Hi

I did read it, but I havn't got to test that part yet. I plan on doing that later on, but for the time being my primary focus have been on optimizing the way the query is generated. And it was in this quest I found the above results.