Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Rasmus Fjord 675 posts 1566 karma points c-trib
    May 27, 2013 @ 10:36
    Rasmus Fjord
    0

    Help, my examine code is acting slow

    Hey guys and girls :)

     

    I have a done a few examine search setups now which is more or likly the same, this time though its really acting up. so a few stats first, its a umb 4.11.6(ill upgrade to 4.11.8 before production but it shouldnt change anything). The index is self consists of :
    Number of Fields: 16
    Number of Documents: 116903
    Number of Terms:310925

    Its the biggest ive worked with so far but what i can read its no biggy.

    So the method im having some trouble with is an autocomplete function that just return 10 items as json. 

    The method looks like this : 

        IEnumerable<SearchResult> advancedSearchResults = new List<SearchResult>();
    
                Dictionary<string, string> searchParamsWithWeights = new Dictionary<string, string>();
                searchParamsWithWeights.Add("Name", "^3");
                searchParamsWithWeights.Add("Group", "1");
                searchParamsWithWeights.Add("Road", "^3");
                searchParamsWithWeights.Add("City", "^1");
                searchParamsWithWeights.Add("Zip", "^1");
                searchParamsWithWeights.Add("Phone", "^2");
                searchParamsWithWeights.Add("Website", "^1");
                searchParamsWithWeights.Add("Email", "^2");
                searchParamsWithWeights.Add("Information", "^1");
                searchParamsWithWeights.Add("MetaWords", "^3");
    
                string luceneString = "+SearchPath:" + parentId + " ";
    
                foreach (var item in searchParamsWithWeights)
                {
                    luceneString += item.Key + ": ";
    
                        luceneString += "(+" + input.Replace(" ", "~0.5 +") + "~0.5)" + item.Value + " ";
    
                }
    
                var query = searchCriteria.RawQuery(luceneString);
                advancedSearchResults = Searcher.Search(query).Where(x => x.Score >= 0.5f);
                advancedSearchResults = advancedSearchResults.OrderBy(x => x.Fields["NodeTypeAlias"] == "faggruppe" ? 1 : 2);
    
                Debug.WriteLine("Time1: " + s.ElapsedMilliseconds);
    
                List<AutocompleteResult> resultList = new List<AutocompleteResult>();
    
    
                foreach (var result in searchResults.Take(10))
                {
                    resultList.Add(new AutocompleteResult { id = result.Id, data = result.Fields["Name"], type = result.Fields["Group"] });
    
                }
    
                if (resultList.Count < 10)
                {
                    Debug.WriteLine("Time2: " + s.ElapsedMilliseconds);
    
                    foreach (var result in advancedSearchResults.Take(10 - resultList.Count))
                    {
                        if (!resultList.Exists(x => x.id == result.Id))
                        {
                            resultList.Add(new AutocompleteResult { id = result.Id, data = result.Fields["Name"], type = result.Fields["Group"] });
    
                            if (resultList.Count == 10)
                            {
                                break;
                            }
                        }
                    }
                }
    Debug.WriteLine("JustBeforeReturn: " + s.ElapsedMilliseconds);
    
    
    return Json.Encode(resultList);

    So ive only pasted some of the code, but the idea is that it does 2 searches, one more specific on the search the next with a lot of fuzzy and looking on a lot of params. 

    Im doing a few debug time traces which looks like this :

    Time1: 456

    Time2: 1263

    JustBeforeReturn: 2011

    So time 1 is being printed when the searches is done and in this search "advancedSearchResults" only contains 35 items or so, so its no biggy. But as soon as we touch the iEnumberable and take 10 items from it the times sky rockets to time 2.

    So it might not be examine but im really wondering why it acting up so much. Ive tried tons of things, anyone has an idea ? There is no rocket sience in the above code.

  • Dan Patching 31 posts 158 karma points c-trib
    May 28, 2013 @ 08:57
    Dan Patching
    100

    Hi Rasmus,

    I think the IEnumerable is the problem, and the Take(10) method is your bottleneck.

    Try using a List or array instead as per the answer to the post below.

    http://stackoverflow.com/questions/3628425/ienumerable-vs-list-what-to-use-how-do-they-work

    IEnumerable describes behavior, while List is an implementation of that behavior. When you use IEnumerable, you give the compiler a chance to defer work until later, possibly optimizing along the way. If you use ToList() you force the compiler to reify the results right away.

    Hope that fixes it. Good luck.

     
  • Rasmus Fjord 675 posts 1566 karma points c-trib
    May 28, 2013 @ 11:25
    Rasmus Fjord
    0

    Hey Dan 

    Thx for leading us in the direction, i think we found a work around for it. 

    A question you might be able to answer aswell ? Is this is a big lucene index ?

  • Dan Patching 31 posts 158 karma points c-trib
    May 28, 2013 @ 11:53
    Dan Patching
    0

    Personally I haven't indexed that many nodes in an Umbraco project before.

    I have heard of people with 300k nodes in Umbraco with no issues.

    There are articles in Google showing experiences of Lucene (without Umbraco) with much larger data sets (1 billion+).

    Lucene should be fine with your data set.

  • Rasmus Fjord 675 posts 1566 karma points c-trib
    May 28, 2013 @ 12:08
    Rasmus Fjord
    0

    Arh okay :)

    Thx for your comments

    its not nodes, its a custom indexer that takes some extra data and such. 

Please Sign in or register to post replies

Write your reply to:

Draft