Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Jason Espin 368 posts 1335 karma points
    Apr 30, 2015 @ 15:53
    Jason Espin
    0

    Advanced multi-stage queries using Examine

    Hi all,

    We are having a lot of issues trying to build some Dynamic queries in Examine. We have a search form that contains the following fields;

    public class TourFinderModel
    {
        public string[] Trip_Type { get; set;}
    
        [DataType(DataType.Date)]
        public DateTime? Date_From { get; set;}
    
        [DataType(DataType.Date)]
        public DateTime? Date_To { get; set; }
    
        public string[] Trip_Duration { get; set; }
    
        public string[] Trip_Difficulty { get; set; }
    }
    

    TripType, TripDuration and Trip_Difficulty can be multi-select and must have at least one value.

    We pass our model to a controller that ensures that the data format is correct based upon the current culture and if this is not the case then we default the datefrom to the current date and the dateto to a date that is 28 days in the future.

    We then convert the TripType, TripDuration and Trip_Difficulty to CSV strings containing the search terms we need to process and pass these to our search page via TempData (see below):

       public class TourFinderController : SurfaceController
            {
                [HttpPost]
                public ActionResult Submit(TourFinderModel model)
                {
                    IPublishedContent page = Umbraco.TypedContent(UmbracoContext.Current.PageId);
                    IPublishedContent homepage = page.AncestorOrSelf(1);
                    IPublishedContent search = homepage.Descendant("search");
    
                    // Set the culture of the page the request came from
                    CultureInfo ci = CultureInfo.GetCultureInfo(page.GetCulture().ToString());
    
                    // Ensure dateTime values have a value
                    DateTime dateFrom = new DateTime();
                    DateTime dateTo = new DateTime();
    
                    if (DateTime.TryParse(model.Date_From.ToString(), ci, DateTimeStyles.None, out dateFrom))
                    {
                        model.Date_From = dateFrom;
                    }
                    else
                    {
                        model.Date_From = DateTime.Now;
                    }
    
                    if (DateTime.TryParse(model.Date_To.ToString(), ci, DateTimeStyles.None, out dateTo))
                    {
                        model.Date_To = dateTo;
                    }
                    else
                    {
                        model.Date_To = dateFrom.AddDays(28);
                    }
    
                    string csvType = string.Empty;
                    string csvDuration = string.Empty;
                    string csvDifficulty = string.Empty;
    
                    csvType = string.Join(",", model.Trip_Type);
                    csvDuration = string.Join(",", model.Trip_Duration);
                    csvDifficulty = string.Join(",", model.Trip_Difficulty);
    
                    // Validate the model
                    if (!ModelState.IsValid)
                    {
                        return CurrentUmbracoPage();
                    }
    
                    TempData["type"] = csvType;
                    TempData["from"] = model.Date_From;
                    TempData["to"] = model.Date_To;
                    TempData["duration"] = csvDuration;
                    TempData["difficulty"] = csvDifficulty;
                    TempData["culture"] = ci;
    
                    return RedirectToUmbracoPage(search);
                }
            }
    

    It is on our search page that we process the search logic based upon the TempData values. As you can see we are trying to build our query dynamically so that we start with one of the criteria and use an Or to specify that for that criteria we are accepting multiple values. For the next value we do an and and then specify the field as well as the multiple values we accept. We then do the same for the last. The dates we leave our of Examine and process using Umbraco because of the way in which they are stored.

    @inherits Umbraco.Web.Mvc.UmbracoTemplatePage
    @using Examine.SearchCriteria
    @using System.Globalization
    @using System.Configuration
    @{
        Layout = "Site.cshtml";
    
        /* Posted Data */
        string difficulty = TempData["difficulty"].ToString();
        string type = TempData["type"].ToString();
        string duration = TempData["duration"].ToString();
        string startDate = TempData["from"].ToString();
        string endDate = TempData["to"].ToString();
    
        Dictionary<string, List<string>> values = new Dictionary<string, List<string>>();
    
    
        if (!string.IsNullOrWhiteSpace(type))
        {
            List<string> csvList = type.Split(',').ToList();     
            values.Add("tripType", csvList);
    
        }
        if (!string.IsNullOrWhiteSpace(difficulty))
        {
            List<string> csvList = difficulty.Split(',').ToList();
            values.Add("tripDifficulty", csvList);
        }
        if (!string.IsNullOrWhiteSpace(duration))
        {
            List<string> csvList = duration.Split(',').ToList();
            values.Add("tripDuration", csvList);
        }
    
        DateTime dateFrom = new DateTime();
        DateTime dateTo = new DateTime();
    
        CultureInfo searchCulture = new CultureInfo(TempData["culture"].ToString());
        CultureInfo databaseCulture = new CultureInfo("en-GB");
    
        DateTime.TryParse(startDate, searchCulture, DateTimeStyles.None, out dateFrom);
        DateTime.TryParse(endDate, searchCulture, DateTimeStyles.None, out dateTo);
    
        List<string> dates = new List<string>();
    
        for (DateTime dt = dateFrom; dt <= dateTo; dt = dt.AddDays(1))
        {
            string date = dt.Date.ToString("dd/MM/yyyy", databaseCulture);
            dates.Add(date);
        }
    
        var searcher = Examine.ExamineManager.Instance.SearchProviderCollection["ToursSearcher"];
        var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.Or);
    
        IBooleanOperation query = null;
    
        if (values.Count() > 0)
        {        
            foreach (KeyValuePair<string, List<string>> entry in values)
            {
                if (query == null)
                {
                    int count = 0;
                    foreach (string value in entry.Value)
                    {
                        if(count == 0){
                            query = searchCriteria.Field(entry.Key, value);
                        }else{
                            query = query.Or().Field(entry.Key, value);
                        }
                        count++;
                    }
                }
                else
                {
                    int count = 0;
                    foreach (string value in entry.Value)
                    {
                        if (count == 0)
                        {
                            query = query.And().Field(entry.Key, value);
                        }
                        else
                        {
                            query = query.Or().Field(entry.Key, value);
                        }
                        count++;
                    }
                }
            }
        }
    
        List<dynamic> umbracoPages = new List<dynamic>();
    
        var searchResults = searcher.Search(query.Compile()).OrderByDescending(x => x.Score).TakeWhile(x => x.Score > 0.5f);
        if (searchResults.Count() > 0)
        {
            foreach (var item in searchResults)
            {
                IPublishedContent node = Umbraco.Content(item.Fields["id"]);
                if (node.GetCulture().ToString() == searchCulture.ToString())
                {
                    List<string> departures = node.GetPropertyValue<string>("departureDate").Split(',').ToList();
    
                    foreach (var date in dates)
                    {
                        if (departures.Contains(date))
                        {
                            var anonObject = new { Difficulty = node.GetPropertyValue("tripDifficulty"), Name = node.Name, Date = date };
                            umbracoPages.Add(anonObject);
                        }
                    }
                }
            }
        }
    }
    
    
    <div class="row">
    
            Number of results : @umbracoPages.Count()
    
    
            <div id="search-results">
                Sort by: 
                <select>
                    <option class="sort" data-sort="name">Name</option>
                    <option class="sort" data-sort="date">Date</option>
                </select>
               <!--<button >
                    Sort by name
                </button>
                <button class="sort" data-sort="date">
                    Sort by date
                </button>
                   --> 
                <div class="list col-md-12">
    
                    @foreach (var page in umbracoPages)
                    {
                        <div class="package col-md-8">
    
                                <span class="name">@page.Name</span><br/>
                                <span class="difficulty">@page.Difficulty</span><br />
                                <span class="date">@page.Date</span>
                        </div>
    
                    }
                </div>
    
            </div>
    </div>
    

    The main issue we have is that our query is not returning any results when really it should.

    We have selected three values for each of the string criteria knowing that one of the values of these three will definitely return a result but nothing is happening.

    What we are aiming for is something like this in examine:

     if((Trip_Type == "valueA" || Trip_Type == "valueB" || Trip_Type == "valueC" ) && (Trip_Duration == "valueD" || Trip_Duration == "valueE" || Trip_Duration == "valueF" ) && (Trip_Difficulty == "valueG" || Trip_Difficulty == "valueH" || Trip_Difficulty == "valueI" ))
    

    So in this case we could have results with:

    Trip_Type = valueB
    Trip_Duration = valueD
    Trip_Difficulty = valueH
    

    Any help with this would be greatly appreciated.

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    Apr 30, 2015 @ 16:30
    Ismail Mayat
    0

    Couple of things, firstly can you on the results page do a searchCriteria.ToString() just before you do the search this will give you a generated lucene query. You then need to extract out of that manually just the query bit as there is other debug info. Next download luke and then open your index and run the query there. Also using luke have a look at how the dates are stored by that i mean format it should be 20150430124423

    On another point not sure why are are ordering by score as it orders in lucene by score already. 

    Regards

    Ismail

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    Apr 30, 2015 @ 16:32
    Ismail Mayat
    0

    Sorry ignore that date bit as I can see you are filtering after search.  However you still need to paste back your generated query and test that in luke.

  • Jason Espin 368 posts 1335 karma points
    Apr 30, 2015 @ 18:05
    Jason Espin
    0

    We are running it in Luke but there seems to be an issue with the way in which Examine is indexing our fields. For fields where we have a phrase it seems to only be matching on the first word despite us using "" in our Lucene query.

    Any ideas as to why this may be happening?

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    Apr 30, 2015 @ 18:11
    Ismail Mayat
    0

    Can you paste the query please.

  • Anne Marie 1 post 21 karma points
    Apr 30, 2015 @ 18:26
    Anne Marie
    0

    Certainly, the query that we construct and pass into searcher.Search() is :

        +tripType:(\"hiking tours\" OR \"expeditions\") + tripDifficulty:(\"easy\" OR \"moderate\") +tripDuration(\"2-5 days\")

    This should only return two results but is returning three. The third does not have a trip type of "expeditions" or "hiking tours" but it does have a type of "private tours" so I believe it is matching on the tour property. All of the other properties match.

    For fields like tripType, tripDifficulty and tripDuration it is possible for there to be multiple values as they are stored in CSV format but that isn't what is causing the issue here.

    We are using the following indexer:

    Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net
    
  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    Apr 30, 2015 @ 18:54
    Ismail Mayat
    0

    Anne,

    So in luke it is also returning three? Looking at the query its phrase so "hiking tours" is a phrase match so it should not match just the word tours.

    Regards

    Ismail

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 00:51
    Jason Espin
    0

    Unfortunately thats not the way it is working.

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    May 01, 2015 @ 09:05
    Ismail Mayat
    0

    Jason,

    Your indexer is standard what about your searcher is that also standard? When you are searching in luke are you using standard as well?

    Regards

    Ismail

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    May 01, 2015 @ 10:38
    Ismail Mayat
    0

    Jason,

    I have been having a play with luke and an index on an existing site,

    +faqAnswer:(\"brands worldwide\" OR \"dish washer\")

    lucene rewrites the query as 

    +(faqAnswer:brands faqAnswer:worldwide faqAnswer:dish faqAnswer:washer)

    I then in the query removed \ and tried again so my query is 

    +faqAnswer:("brands worldwide" OR "dish washer")

    this rewrites to

    +(faqAnswer:"brands worldwide" faqAnswer:"dish washer")

    now that is definately a phrase query. In luke repaste your query remove \ and see if that works.

    Regards

    Ismail

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    May 01, 2015 @ 10:39
    Ismail Mayat
    0

    Also can you try one more thing in your code 

    var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.Or);

    Change that to BooleanOperation.And see what that does.

    Regards

    Ismail

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 11:57
    Jason Espin
    0

    Hi Ismail,

    He is what we have in our ExamineSettings.config:

    <add name="ToursIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
               supportUnpublished="false"
               supportProtected="false"
               analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
               indexSet="Tours"/>
    
    <add name="ToursSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
               analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" indexSet="Tours" enableLeadingWildcard="true"/>
    

    When we are searching in Luke we are using the same indexer. The problem we face is in Luke we are able to construct the queries and get back what we expect apart from with the tripDuration field which has a value of "2-5 days". For some reason Luke reformats this into "2 5 days" which does not match what is in the index.

    Regardless of this, when we search the index in Umbraco we are not getting back what we should.

    Kind regards,

    Jason Espin

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    May 01, 2015 @ 12:03
    Ismail Mayat
    0

    Jason,

    Did you try using And as default criteria, it looks to me as though the escaping character which is in the generated query is resulting in a non phrase query. With regards to - its a special character see https://lucene.apache.org/core/2_9_4/queryparsersyntax.html

    Regards

    Ismail

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 12:18
    Jason Espin
    0

    Hi Ismail,

    We have tried using And as out default criteria and this doesn't make any difference. We have also tried changing our criteria now so whereas before the tripType was "Hiking Tours" we now just use an int like 1 however despite only having one tour with 1 it is returning other tours as well.

    This is the whole of our search logic:

    @inherits Umbraco.Web.Mvc.UmbracoTemplatePage
    @using Examine.SearchCriteria
    @using System.Globalization
    @using System.Configuration
    @{
        Layout = "Site.cshtml";
    
        /* Posted Data */
        string difficulty = TempData["difficulty"].ToString();
        string type = TempData["type"].ToString();
        string duration = TempData["duration"].ToString();
        string startDate = TempData["from"].ToString();
        string endDate = TempData["to"].ToString();
    
        Dictionary<string, List<string>> values = new Dictionary<string, List<string>>();
    
    
        if (!string.IsNullOrWhiteSpace(type))
        {
            List<string> csvList = type.Split(',').ToList();     
            values.Add("tripType", csvList);
    
        }
        if (!string.IsNullOrWhiteSpace(difficulty))
        {
            List<string> csvList = difficulty.Split(',').ToList();
            values.Add("tripDifficulty", csvList);
        }
        if (!string.IsNullOrWhiteSpace(duration))
        {
            List<string> csvList = duration.Split(',').ToList();
            values.Add("tripDuration", csvList);
        }
    
        DateTime dateFrom = new DateTime();
        DateTime dateTo = new DateTime();
    
        CultureInfo searchCulture = new CultureInfo(TempData["culture"].ToString());
        CultureInfo databaseCulture = new CultureInfo("en-GB");
    
        DateTime.TryParse(startDate, searchCulture, DateTimeStyles.None, out dateFrom);
        DateTime.TryParse(endDate, searchCulture, DateTimeStyles.None, out dateTo);
    
        List<string> dates = new List<string>();
    
        for (DateTime dt = dateFrom; dt <= dateTo; dt = dt.AddDays(1))
        {
            string date = dt.Date.ToString("dd/MM/yyyy", databaseCulture);
            dates.Add(date);
        }
    
        var searcher = Examine.ExamineManager.Instance.SearchProviderCollection["ToursSearcher"];
        var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.Or);
    
        IBooleanOperation query = null;
    
        int count = 0;
        string queryValue = string.Empty;
        string rawQuery = string.Empty;
    
        if (values.Count() > 0)
        {        
            foreach (KeyValuePair<string, List<string>> entry in values)
            {
                    queryValue = string.Empty;
                    foreach (string value in entry.Value)
                    {
                        queryValue += string.Format(@"""{0}"" OR ", value.ToLower());
                    }
    
                    queryValue = "(" + queryValue.TrimEnd(" OR ") + ")";
                    //query = searchCriteria.Field(entry.Key, queryValue);
                    rawQuery += string.Format(" +{0}:{1}", entry.Key, queryValue);
            }
            rawQuery = rawQuery.Trim();
            rawQuery.Replace(@"""", string.Empty);
    
    
        }
    
        List<dynamic> umbracoPages = new List<dynamic>();
    
        //var g = query.Compile();
    
        var searchResults = searcher.Search(rawQuery,false);
        if (searchResults.Count() > 0)
        {
            foreach (var item in searchResults)
            {
                IPublishedContent node = Umbraco.Content(item.Fields["id"]);
                if (node.GetCulture().ToString() == searchCulture.ToString())
                {
    
                    // Add logic for empty dates
                    List<string> departures = node.GetPropertyValue<string>("departureDate").Split(',').ToList();
    
                    foreach (var date in dates)
                    {
                        if (departures.Contains(date))
                        {
                            var anonObject = new { Difficulty = node.GetPropertyValue("tripDifficulty"), Name = node.Name, Date = date };
                            umbracoPages.Add(anonObject);
                        }
                    }
                }
            }
        }
    }
    
    
    <div class="row">
    
            Number of results : @umbracoPages.Count()
    
    
            <div id="search-results">
                Sort by: 
                <select>
                    <option class="sort" data-sort="name">Name</option>
                    <option class="sort" data-sort="date">Date</option>
                </select>
               <!--<button >
                    Sort by name
                </button>
                <button class="sort" data-sort="date">
                    Sort by date
                </button>
                   --> 
                <div class="list col-md-12">
    
                    @foreach (var page in umbracoPages)
                    {
                        <div class="package col-md-8">
    
                                <span class="name">@page.Name</span><br/>
                                <span class="difficulty">@page.Difficulty</span><br />
                                <span class="date">@page.Date</span>
    
                        </div>
    
                    }
                </div>
    
            </div>
    </div>
    

    And these are the values we are passing in:

    string difficulty = "easy"; /* This can be a CSV list */
    string type = "1"; /* This can be a CSV list */
    string duration = "2-5 days"; /* This can be a CSV list */
    /* Dates are parsed outside of Lucene */
    

    This basic combination should only return one result however it is returning 3 one of which has difficulty moderate and also has a type other than 1

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 12:42
    Jason Espin
    0

    We are now desperately trying simple queries as Examine just doesn't seem to work consistently or correctly. So, I have entered the following in the back office of Umbraco as a Lucene query:

    +tripDifficulty:easy

    This correctly returns 4 results. When I do this in Visual Studio:

    string ourQuery = "+tripDifficulty:easy";
    
    var searchResults = searcher.Search(ourQuery,false);
    

    Nothing is returned.

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    May 01, 2015 @ 13:19
    Ismail Mayat
    0

    Jason,

    For a raw query you have todo searcher.RawQuery so it knows that the query is direct lucene query else it wont work.

    Regards

    Ismail

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 13:25
    Jason Espin
    0

    Hi Ismail,

    We figured that out just before your post however it is still not returning what we would expect.

    Kind regards,

    Jason Espin

  • Ismail Mayat 4511 posts 10091 karma points MVP 2x admin c-trib
    May 01, 2015 @ 13:42
    Ismail Mayat
    0

    Ideally i need to see whats going on you on skype? Im on ismail_mayat can take a looksie after 2 today if your free

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 13:54
    Jason Espin
    0

    We have tried multiple combinations now when using searchCriteria.RawQuery(rawQuery). These are the compiled queries that are produced:

    /* BooleanOperation.Or */
    SearchIndexType: "", LuceneQuery: {(+tripType:2 +tripDifficulty:easy +tripDuration:"2-5 days")}
    
    /* BooleanOperation.And */
    SearchIndexType: "", LuceneQuery: {+(+tripType:2 +tripDifficulty:easy +tripDuration:"2-5 days")}
    
    /* No Boolean operation specified */
    SearchIndexType: "", LuceneQuery: {+(+tripType:2 +tripDifficulty:easy +tripDuration:"2-5 days")}
    

    We know for a fact that there is one record that matches this criteria in Umbraco:

    enter image description here

    So, if this was working correctly, this record would be returned. However it is not.

  • Jason Espin 368 posts 1335 karma points
    May 01, 2015 @ 13:56
    Jason Espin
    0

    Hi Ismail,

    I am free after 2 today also and your help is greatly appreciated. I will add you on Skype shortly.

    What time is best for you?

    Thanks again for all of your hel so far.

Please Sign in or register to post replies

Write your reply to:

Draft