We are having a lot of issues trying to build some Dynamic queries in Examine.
We have a search form that contains the following fields;
public class TourFinderModel
{
public string[] Trip_Type { get; set;}
[DataType(DataType.Date)]
public DateTime? Date_From { get; set;}
[DataType(DataType.Date)]
public DateTime? Date_To { get; set; }
public string[] Trip_Duration { get; set; }
public string[] Trip_Difficulty { get; set; }
}
TripType, TripDuration and Trip_Difficulty can be multi-select and must have at least one value.
We pass our model to a controller that ensures that the data format is correct based upon the current culture and if this is not the case then we default the datefrom to the current date and the dateto to a date that is 28 days in the future.
We then convert the TripType, TripDuration and Trip_Difficulty to CSV strings containing the search terms we need to process and pass these to our search page via TempData (see below):
public class TourFinderController : SurfaceController
{
[HttpPost]
public ActionResult Submit(TourFinderModel model)
{
IPublishedContent page = Umbraco.TypedContent(UmbracoContext.Current.PageId);
IPublishedContent homepage = page.AncestorOrSelf(1);
IPublishedContent search = homepage.Descendant("search");
// Set the culture of the page the request came from
CultureInfo ci = CultureInfo.GetCultureInfo(page.GetCulture().ToString());
// Ensure dateTime values have a value
DateTime dateFrom = new DateTime();
DateTime dateTo = new DateTime();
if (DateTime.TryParse(model.Date_From.ToString(), ci, DateTimeStyles.None, out dateFrom))
{
model.Date_From = dateFrom;
}
else
{
model.Date_From = DateTime.Now;
}
if (DateTime.TryParse(model.Date_To.ToString(), ci, DateTimeStyles.None, out dateTo))
{
model.Date_To = dateTo;
}
else
{
model.Date_To = dateFrom.AddDays(28);
}
string csvType = string.Empty;
string csvDuration = string.Empty;
string csvDifficulty = string.Empty;
csvType = string.Join(",", model.Trip_Type);
csvDuration = string.Join(",", model.Trip_Duration);
csvDifficulty = string.Join(",", model.Trip_Difficulty);
// Validate the model
if (!ModelState.IsValid)
{
return CurrentUmbracoPage();
}
TempData["type"] = csvType;
TempData["from"] = model.Date_From;
TempData["to"] = model.Date_To;
TempData["duration"] = csvDuration;
TempData["difficulty"] = csvDifficulty;
TempData["culture"] = ci;
return RedirectToUmbracoPage(search);
}
}
It is on our search page that we process the search logic based upon the TempData values. As you can see we are trying to build our query dynamically so that we start with one of the criteria and use an Or to specify that for that criteria we are accepting multiple values. For the next value we do an and and then specify the field as well as the multiple values we accept. We then do the same for the last. The dates we leave our of Examine and process using Umbraco because of the way in which they are stored.
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage
@using Examine.SearchCriteria
@using System.Globalization
@using System.Configuration
@{
Layout = "Site.cshtml";
/* Posted Data */
string difficulty = TempData["difficulty"].ToString();
string type = TempData["type"].ToString();
string duration = TempData["duration"].ToString();
string startDate = TempData["from"].ToString();
string endDate = TempData["to"].ToString();
Dictionary<string, List<string>> values = new Dictionary<string, List<string>>();
if (!string.IsNullOrWhiteSpace(type))
{
List<string> csvList = type.Split(',').ToList();
values.Add("tripType", csvList);
}
if (!string.IsNullOrWhiteSpace(difficulty))
{
List<string> csvList = difficulty.Split(',').ToList();
values.Add("tripDifficulty", csvList);
}
if (!string.IsNullOrWhiteSpace(duration))
{
List<string> csvList = duration.Split(',').ToList();
values.Add("tripDuration", csvList);
}
DateTime dateFrom = new DateTime();
DateTime dateTo = new DateTime();
CultureInfo searchCulture = new CultureInfo(TempData["culture"].ToString());
CultureInfo databaseCulture = new CultureInfo("en-GB");
DateTime.TryParse(startDate, searchCulture, DateTimeStyles.None, out dateFrom);
DateTime.TryParse(endDate, searchCulture, DateTimeStyles.None, out dateTo);
List<string> dates = new List<string>();
for (DateTime dt = dateFrom; dt <= dateTo; dt = dt.AddDays(1))
{
string date = dt.Date.ToString("dd/MM/yyyy", databaseCulture);
dates.Add(date);
}
var searcher = Examine.ExamineManager.Instance.SearchProviderCollection["ToursSearcher"];
var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.Or);
IBooleanOperation query = null;
if (values.Count() > 0)
{
foreach (KeyValuePair<string, List<string>> entry in values)
{
if (query == null)
{
int count = 0;
foreach (string value in entry.Value)
{
if(count == 0){
query = searchCriteria.Field(entry.Key, value);
}else{
query = query.Or().Field(entry.Key, value);
}
count++;
}
}
else
{
int count = 0;
foreach (string value in entry.Value)
{
if (count == 0)
{
query = query.And().Field(entry.Key, value);
}
else
{
query = query.Or().Field(entry.Key, value);
}
count++;
}
}
}
}
List<dynamic> umbracoPages = new List<dynamic>();
var searchResults = searcher.Search(query.Compile()).OrderByDescending(x => x.Score).TakeWhile(x => x.Score > 0.5f);
if (searchResults.Count() > 0)
{
foreach (var item in searchResults)
{
IPublishedContent node = Umbraco.Content(item.Fields["id"]);
if (node.GetCulture().ToString() == searchCulture.ToString())
{
List<string> departures = node.GetPropertyValue<string>("departureDate").Split(',').ToList();
foreach (var date in dates)
{
if (departures.Contains(date))
{
var anonObject = new { Difficulty = node.GetPropertyValue("tripDifficulty"), Name = node.Name, Date = date };
umbracoPages.Add(anonObject);
}
}
}
}
}
}
<div class="row">
Number of results : @umbracoPages.Count()
<div id="search-results">
Sort by:
<select>
<option class="sort" data-sort="name">Name</option>
<option class="sort" data-sort="date">Date</option>
</select>
<!--<button >
Sort by name
</button>
<button class="sort" data-sort="date">
Sort by date
</button>
-->
<div class="list col-md-12">
@foreach (var page in umbracoPages)
{
<div class="package col-md-8">
<span class="name">@page.Name</span><br/>
<span class="difficulty">@page.Difficulty</span><br />
<span class="date">@page.Date</span>
</div>
}
</div>
</div>
</div>
The main issue we have is that our query is not returning any results when really it should.
We have selected three values for each of the string criteria knowing that one of the values of these three will definitely return a result but nothing is happening.
What we are aiming for is something like this in examine:
Couple of things, firstly can you on the results page do a searchCriteria.ToString() just before you do the search this will give you a generated lucene query. You then need to extract out of that manually just the query bit as there is other debug info. Next download luke and then open your index and run the query there. Also using luke have a look at how the dates are stored by that i mean format it should be 20150430124423
On another point not sure why are are ordering by score as it orders in lucene by score already.
Sorry ignore that date bit as I can see you are filtering after search. However you still need to paste back your generated query and test that in luke.
We are running it in Luke but there seems to be an issue with the way in which Examine is indexing our fields. For fields where we have a phrase it seems to only be matching on the first word despite us using "" in our Lucene query.
Certainly, the query that we construct and pass into searcher.Search() is :
+tripType:(\"hiking tours\" OR \"expeditions\") + tripDifficulty:(\"easy\" OR \"moderate\") +tripDuration(\"2-5 days\")
This should only return two results but is returning three. The third does not have a trip type of "expeditions" or "hiking tours" but it does have a type of "private tours" so I believe it is matching on the tour property. All of the other properties match.
For fields like tripType, tripDifficulty and tripDuration it is possible for there to be multiple values as they are stored in CSV format but that isn't what is causing the issue here.
When we are searching in Luke we are using the same indexer. The problem we face is in Luke we are able to construct the queries and get back what we expect apart from with the tripDuration field which has a value of "2-5 days". For some reason Luke reformats this into "2 5 days" which does not match what is in the index.
Regardless of this, when we search the index in Umbraco we are not getting back what we should.
Did you try using And as default criteria, it looks to me as though the escaping character which is in the generated query is resulting in a non phrase query. With regards to - its a special character see https://lucene.apache.org/core/2_9_4/queryparsersyntax.html
We have tried using And as out default criteria and this doesn't make any difference.
We have also tried changing our criteria now so whereas before the tripType was "Hiking Tours" we now just use an int like 1 however despite only having one tour with 1 it is returning other tours as well.
This is the whole of our search logic:
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage
@using Examine.SearchCriteria
@using System.Globalization
@using System.Configuration
@{
Layout = "Site.cshtml";
/* Posted Data */
string difficulty = TempData["difficulty"].ToString();
string type = TempData["type"].ToString();
string duration = TempData["duration"].ToString();
string startDate = TempData["from"].ToString();
string endDate = TempData["to"].ToString();
Dictionary<string, List<string>> values = new Dictionary<string, List<string>>();
if (!string.IsNullOrWhiteSpace(type))
{
List<string> csvList = type.Split(',').ToList();
values.Add("tripType", csvList);
}
if (!string.IsNullOrWhiteSpace(difficulty))
{
List<string> csvList = difficulty.Split(',').ToList();
values.Add("tripDifficulty", csvList);
}
if (!string.IsNullOrWhiteSpace(duration))
{
List<string> csvList = duration.Split(',').ToList();
values.Add("tripDuration", csvList);
}
DateTime dateFrom = new DateTime();
DateTime dateTo = new DateTime();
CultureInfo searchCulture = new CultureInfo(TempData["culture"].ToString());
CultureInfo databaseCulture = new CultureInfo("en-GB");
DateTime.TryParse(startDate, searchCulture, DateTimeStyles.None, out dateFrom);
DateTime.TryParse(endDate, searchCulture, DateTimeStyles.None, out dateTo);
List<string> dates = new List<string>();
for (DateTime dt = dateFrom; dt <= dateTo; dt = dt.AddDays(1))
{
string date = dt.Date.ToString("dd/MM/yyyy", databaseCulture);
dates.Add(date);
}
var searcher = Examine.ExamineManager.Instance.SearchProviderCollection["ToursSearcher"];
var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.Or);
IBooleanOperation query = null;
int count = 0;
string queryValue = string.Empty;
string rawQuery = string.Empty;
if (values.Count() > 0)
{
foreach (KeyValuePair<string, List<string>> entry in values)
{
queryValue = string.Empty;
foreach (string value in entry.Value)
{
queryValue += string.Format(@"""{0}"" OR ", value.ToLower());
}
queryValue = "(" + queryValue.TrimEnd(" OR ") + ")";
//query = searchCriteria.Field(entry.Key, queryValue);
rawQuery += string.Format(" +{0}:{1}", entry.Key, queryValue);
}
rawQuery = rawQuery.Trim();
rawQuery.Replace(@"""", string.Empty);
}
List<dynamic> umbracoPages = new List<dynamic>();
//var g = query.Compile();
var searchResults = searcher.Search(rawQuery,false);
if (searchResults.Count() > 0)
{
foreach (var item in searchResults)
{
IPublishedContent node = Umbraco.Content(item.Fields["id"]);
if (node.GetCulture().ToString() == searchCulture.ToString())
{
// Add logic for empty dates
List<string> departures = node.GetPropertyValue<string>("departureDate").Split(',').ToList();
foreach (var date in dates)
{
if (departures.Contains(date))
{
var anonObject = new { Difficulty = node.GetPropertyValue("tripDifficulty"), Name = node.Name, Date = date };
umbracoPages.Add(anonObject);
}
}
}
}
}
}
<div class="row">
Number of results : @umbracoPages.Count()
<div id="search-results">
Sort by:
<select>
<option class="sort" data-sort="name">Name</option>
<option class="sort" data-sort="date">Date</option>
</select>
<!--<button >
Sort by name
</button>
<button class="sort" data-sort="date">
Sort by date
</button>
-->
<div class="list col-md-12">
@foreach (var page in umbracoPages)
{
<div class="package col-md-8">
<span class="name">@page.Name</span><br/>
<span class="difficulty">@page.Difficulty</span><br />
<span class="date">@page.Date</span>
</div>
}
</div>
</div>
</div>
And these are the values we are passing in:
string difficulty = "easy"; /* This can be a CSV list */
string type = "1"; /* This can be a CSV list */
string duration = "2-5 days"; /* This can be a CSV list */
/* Dates are parsed outside of Lucene */
This basic combination should only return one result however it is returning 3 one of which has difficulty moderate and also has a type other than 1
We are now desperately trying simple queries as Examine just doesn't seem to work consistently or correctly.
So, I have entered the following in the back office of Umbraco as a Lucene query:
+tripDifficulty:easy
This correctly returns 4 results. When I do this in Visual Studio:
string ourQuery = "+tripDifficulty:easy";
var searchResults = searcher.Search(ourQuery,false);
Advanced multi-stage queries using Examine
Hi all,
We are having a lot of issues trying to build some Dynamic queries in Examine. We have a search form that contains the following fields;
TripType, TripDuration and Trip_Difficulty can be multi-select and must have at least one value.
We pass our model to a controller that ensures that the data format is correct based upon the current culture and if this is not the case then we default the datefrom to the current date and the dateto to a date that is 28 days in the future.
We then convert the TripType, TripDuration and Trip_Difficulty to CSV strings containing the search terms we need to process and pass these to our search page via TempData (see below):
It is on our search page that we process the search logic based upon the TempData values. As you can see we are trying to build our query dynamically so that we start with one of the criteria and use an Or to specify that for that criteria we are accepting multiple values. For the next value we do an and and then specify the field as well as the multiple values we accept. We then do the same for the last. The dates we leave our of Examine and process using Umbraco because of the way in which they are stored.
The main issue we have is that our query is not returning any results when really it should.
We have selected three values for each of the string criteria knowing that one of the values of these three will definitely return a result but nothing is happening.
What we are aiming for is something like this in examine:
So in this case we could have results with:
Any help with this would be greatly appreciated.
Couple of things, firstly can you on the results page do a searchCriteria.ToString() just before you do the search this will give you a generated lucene query. You then need to extract out of that manually just the query bit as there is other debug info. Next download luke and then open your index and run the query there. Also using luke have a look at how the dates are stored by that i mean format it should be 20150430124423
On another point not sure why are are ordering by score as it orders in lucene by score already.
Regards
Ismail
Sorry ignore that date bit as I can see you are filtering after search. However you still need to paste back your generated query and test that in luke.
We are running it in Luke but there seems to be an issue with the way in which Examine is indexing our fields. For fields where we have a phrase it seems to only be matching on the first word despite us using "" in our Lucene query.
Any ideas as to why this may be happening?
Can you paste the query please.
Certainly, the query that we construct and pass into searcher.Search() is :
+tripType:(\"hiking tours\" OR \"expeditions\") + tripDifficulty:(\"easy\" OR \"moderate\") +tripDuration(\"2-5 days\")
This should only return two results but is returning three. The third does not have a trip type of "expeditions" or "hiking tours" but it does have a type of "private tours" so I believe it is matching on the tour property. All of the other properties match.
For fields like tripType, tripDifficulty and tripDuration it is possible for there to be multiple values as they are stored in CSV format but that isn't what is causing the issue here.
We are using the following indexer:
Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net
Anne,
So in luke it is also returning three? Looking at the query its phrase so "hiking tours" is a phrase match so it should not match just the word tours.
Regards
Ismail
Unfortunately thats not the way it is working.
Jason,
Your indexer is standard what about your searcher is that also standard? When you are searching in luke are you using standard as well?
Regards
Ismail
Jason,
I have been having a play with luke and an index on an existing site,
+faqAnswer:(\"brands worldwide\" OR \"dish washer\")
lucene rewrites the query as
+(faqAnswer:brands faqAnswer:worldwide faqAnswer:dish faqAnswer:washer)
I then in the query removed \ and tried again so my query is
+faqAnswer:("brands worldwide" OR "dish washer")
this rewrites to
+(faqAnswer:"brands worldwide" faqAnswer:"dish washer")
now that is definately a phrase query. In luke repaste your query remove \ and see if that works.
Regards
Ismail
Also can you try one more thing in your code
Change that to BooleanOperation.And see what that does.
Regards
Ismail
Hi Ismail,
He is what we have in our ExamineSettings.config:
When we are searching in Luke we are using the same indexer. The problem we face is in Luke we are able to construct the queries and get back what we expect apart from with the tripDuration field which has a value of "2-5 days". For some reason Luke reformats this into "2 5 days" which does not match what is in the index.
Regardless of this, when we search the index in Umbraco we are not getting back what we should.
Kind regards,
Jason Espin
Jason,
Did you try using And as default criteria, it looks to me as though the escaping character which is in the generated query is resulting in a non phrase query. With regards to - its a special character see https://lucene.apache.org/core/2_9_4/queryparsersyntax.html
Regards
Ismail
Hi Ismail,
We have tried using And as out default criteria and this doesn't make any difference. We have also tried changing our criteria now so whereas before the tripType was "Hiking Tours" we now just use an int like 1 however despite only having one tour with 1 it is returning other tours as well.
This is the whole of our search logic:
And these are the values we are passing in:
This basic combination should only return one result however it is returning 3 one of which has difficulty moderate and also has a type other than 1
We are now desperately trying simple queries as Examine just doesn't seem to work consistently or correctly. So, I have entered the following in the back office of Umbraco as a Lucene query:
This correctly returns 4 results. When I do this in Visual Studio:
Nothing is returned.
Jason,
For a raw query you have todo searcher.RawQuery so it knows that the query is direct lucene query else it wont work.
Regards
Ismail
Hi Ismail,
We figured that out just before your post however it is still not returning what we would expect.
Kind regards,
Jason Espin
Ideally i need to see whats going on you on skype? Im on ismail_mayat can take a looksie after 2 today if your free
We have tried multiple combinations now when using searchCriteria.RawQuery(rawQuery). These are the compiled queries that are produced:
We know for a fact that there is one record that matches this criteria in Umbraco:
So, if this was working correctly, this record would be returned. However it is not.
Hi Ismail,
I am free after 2 today also and your help is greatly appreciated. I will add you on Skype shortly.
What time is best for you?
Thanks again for all of your hel so far.
is working on a reply...