+(+nodeTypeAlias:product +__budgetFrom:[0 TO 101]) +__IndexType:content
when i run the code i get no results. When i put the query through luke i get 2 results. In my index i have 3 product items with budgetFrom (in index property is in as __budgetFrom) the values are 50,100,100 for that query i would expect it to return 3 items both in luke and via code.
Hi Ismail, Lucene indexers should provide either the padding or the field type for you based on the incoming data. Plus, more recent version have a numeric field type which avoids you having to do this kind of padding.
Having chatted with Shannon before about his implementation for Examine, I'm pretty sure he's doing this somewhere. I'll punt this thread Shannon's way and hopefully he can shed some light on it for you, to avoid doing the padding yourself.
Awesome. Given the fact we have IBooleanOperationRange(string fieldName,int start,intend,bool includeLower,bool includeUpper); I would assume that its being handled as well.
Yeah internally we're using NumericRangeQuery to build it up.
What may be a problem is we're not pushing the data into Lucene using any numeric tools. I started looking at this the other day but I got a bit confused (and so far we hadn't had any problem with the current method) and left it alone.
I am getting a Lucene book shortly so I'll have another look once I get a better understanding of how to handle numbers with Lucene ;)
I need to order my copy of lucene in action v2 awesome book used to have v1 at my last work place. For now im gonna pad as i need to get this out the door.
unless aaron has done some padding stuff in Examine, i don't think it is in there. Don't think the range fluentapi has a test written either :( will ak aaron tomorrow.
I have implemented ExamineEvents_GatheringNodeData and know which field aliases are range ones and they are all integers so I have done the following :
I am desperate for help on this issue I am doing a range search and getting results back however some results shouldnt be there case and point:
int Model.MinPrice = 300000
int Model.MaxPrice = 500000
query.And().Range("price", model.MinPrice.ToString("D6"), model.MaxPrice.ToString("D6"));
Is return a results including 50000 which isnt in the range and advise on this appreciated its actually driving me mad. Its for a client could really do with getting this complete.
The 5000 record can you open it up in Luke and paste back how that value is stored in the index. Are using gathering node and storing a d6 value? If not then this maybe the issue.
Goto https://code.google.com/archive/p/luke/downloads download v3.5.0 you will need to install Java as well. After that double click the jar file and you can then point it to your index and browse through records in the index.
One thing did you use gathering node event to format the field you are searching over? I can see you are formatting the range values however the field in the index needs be in same format else you can get in expected results
I have tried implementing the below is this anywhere near close?
private void OnGatheringNodeData(object sender, IndexingNodeDataEventArgs e)
{
Node theNode = new Node(e.NodeId);
//check if this is 'Content' (as opposed to media, etc...)
if (e.IndexType == IndexTypes.Content)
{
if (theNode.NodeTypeAlias == "umbPropertyDetails")
{
var node = new Node(e.NodeId);
var propertyValue = TpayneandCo.App_Code.ExamineEventsHelper.GetPropertyValue(theNode, "price");
if (propertyValue != string.Empty)
{
e.Fields.Add("__" + "price", GetFieldValue(e, propertyValue, "price"));
}
}
AddToContentsField(e);
}
}
private void AddToContentsField(IndexingNodeDataEventArgs e)
{
Dictionary<string, string> fields = e.Fields;
var combinedFields = new StringBuilder();
foreach (KeyValuePair<string, string> keyValuePair in fields)
{
combinedFields.AppendLine(keyValuePair.Value);
}
e.Fields.Add("contents", combinedFields.ToString());
}
private string GetFieldValue(IndexingNodeDataEventArgs e, string propertyValue, string luceneFieldAlias)
{
int nodeId = 0;
int.TryParse(propertyValue, out nodeId);
var n = new Node(nodeId);
//node does not exist but we have numeric value
if (n.Id != 0)
{
//have to pad out to get lucene range queries to work
int i = 0;
int.TryParse(n.Name, out i);
return i.ToString("00000000");
}
return nodeId.ToString("00000000");
}
The field you have highlighted in Luke is not d6 format. In gathering node any field you want to range search format it to d6 as you are doing when querying.
The padded value appears to be getting added but I cant see it in Luke or in developer section on umbraco examine managment tab.
public class ExamineEvents : ApplicationBase
{
public ExamineEvents()
{
//Add event handler for 'GatheringNodeData'
ExamineManager.Instance.IndexProviderCollection["ExternalIndexer"].GatheringNodeData += ExamineEvents_GatheringNodeData;
}
void ExamineEvents_GatheringNodeData(object sender, IndexingNodeDataEventArgs e)
{
Node theNode = new Node(e.NodeId);
//check if this is 'Content' (as opposed to media, etc...)
if (e.IndexType == IndexTypes.Content)
{
if (theNode.NodeTypeAlias == "Patient")
{
//var node = new Node(e.NodeId);
var propertyValue = ExamineEventsHelper.GetPropertyValue(theNode, "price");
if (propertyValue != string.Empty)
{
e.Fields.Add("__" + "paddedPrice", GetFieldValue(e, propertyValue, "price"));
}
}
AddToContentsField(e);
}
}
private void AddToContentsField(IndexingNodeDataEventArgs e)
{
Dictionary<string, string> fields = e.Fields;
var combinedFields = new StringBuilder();
foreach (KeyValuePair<string, string> keyValuePair in fields)
{
combinedFields.AppendLine(keyValuePair.Value);
}
e.Fields.Add("contents", combinedFields.ToString());
}
private string GetFieldValue(IndexingNodeDataEventArgs e, string propertyValue, string luceneFieldAlias)
{
int nodeId = e.NodeId;
//int.TryParse(propertyValue, out nodeId);
var n = new Node(nodeId);
//node does not exist but we have numeric value
if (n.Id != 0)
{
//have to pad out to get lucene range queries to work
int i = 0;
//int.TryParse(n.Name, out i);
int.TryParse(propertyValue, out i);
return i.ToString("D6");
}
return nodeId.ToString("D6");
}
}
ExamineIndex.config
<?xml version="1.0"?>
<!--
Umbraco examine is an extensible indexer and search engine.
This configuration file can be extended to create your own index sets.
Index/Search providers can be defined in the UmbracoSettings.config
More information and documentation can be found on CodePlex: http://umbracoexamine.codeplex.com
-->
<ExamineLuceneIndexSets>
<!-- The internal index set used by Umbraco back-office - DO NOT REMOVE -->
<IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/Internal/"/>
<!-- The internal index set used by Umbraco back-office for indexing members - DO NOT REMOVE -->
<IndexSet SetName="InternalMemberIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/InternalMember/">
<IndexAttributeFields>
<add Name="id" />
<add Name="nodeName"/>
<add Name="updateDate" />
<add Name="writerName" />
<add Name="loginName" />
<add Name="email" />
<add Name="nodeTypeAlias" />
</IndexAttributeFields>
</IndexSet>
<!-- Default Indexset for external searches, this indexes all fields on all types of nodes-->
<IndexSet SetName="ExternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/External/">
<IndexAttributeFields>
<add Name="id" />
<add Name="nodeName"/>
<add Name="nodeTypeAlias" />
<add Name="parentID" />
</IndexAttributeFields>
<IndexUserFields/>
<IncludeNodeTypes/>
<ExcludeNodeTypes />
</IndexSet>
</ExamineLuceneIndexSets>
ExamineSettings.config:
<?xml version="1.0"?>
<!--
Umbraco examine is an extensible indexer and search engine.
This configuration file can be extended to add your own search/index providers.
Index sets can be defined in the ExamineIndex.config if you're using the standard provider model.
More information and documentation can be found on CodePlex: http://umbracoexamine.codeplex.com
-->
<Examine>
<ExamineIndexProviders>
<providers>
<add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
supportUnpublished="true"
supportProtected="true"
analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>
<add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
supportUnpublished="true"
supportProtected="true"
analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"/>
<!-- default external indexer, which excludes protected and unpublished pages-->
<add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
runAsync="true"
supportUnpublished="false"
supportProtected="true"
interval="10"
analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
indexSet="ExternalIndexSet"/>
</providers>
</ExamineIndexProviders>
<ExamineSearchProviders defaultProvider="ExternalSearcher">
<providers>
<add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>
<add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
indexSet="ExternalIndexSet"/>
<add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"/>
</providers>
</ExamineSearchProviders>
</Examine>
Yes on both counts rebuilt the index using Umbraco developer section and stepped through and I can see __paddedPrice getting added and formatted correctly but unfortunately still not appaearing in luke or index.
Seem to be getting added to the content field but im expecting another field __paddedPrice
Examine range query numeric
I have a range query it spits out lucene query:
+(+nodeTypeAlias:product +__budgetFrom:[0 TO 101]) +__IndexType:content
when i run the code i get no results. When i put the query through luke i get 2 results. In my index i have 3 product items with budgetFrom (in index property is in as __budgetFrom) the values are 50,100,100 for that query i would expect it to return 3 items both in luke and via code.
Is it treating that numeric value as string? see http://wiki.apache.org/lucene-java/SearchNumericalFields
Regards
Ismail
Right in answer to my question it does treat as string you need to pad out when adding to index see http://stackoverflow.com/questions/708075/problem-using-lucene-rangequery
Ok another update i now can get results same as luke i changed my code from
IBooleanOperation Range(string fieldName, int start, int end, bool includeLower, bool includeUpper);
to
IBooleanOperation Range(string fieldName, string start, string end);
So just need to do the padding thing and should be onto a winner!
Hi Ismail, Lucene indexers should provide either the padding or the field type for you based on the incoming data. Plus, more recent version have a numeric field type which avoids you having to do this kind of padding.
Having chatted with Shannon before about his implementation for Examine, I'm pretty sure he's doing this somewhere. I'll punt this thread Shannon's way and hopefully he can shed some light on it for you, to avoid doing the padding yourself.
Alex,
Awesome. Given the fact we have IBooleanOperation Range(string fieldName, int start, int end, bool includeLower, bool includeUpper); I would assume that its being handled as well.
Regards
Ismail
Yeah internally we're using NumericRangeQuery to build it up.
What may be a problem is we're not pushing the data into Lucene using any numeric tools. I started looking at this the other day but I got a bit confused (and so far we hadn't had any problem with the current method) and left it alone.
I am getting a Lucene book shortly so I'll have another look once I get a better understanding of how to handle numbers with Lucene ;)
I need to order my copy of lucene in action v2 awesome book used to have v1 at my last work place. For now im gonna pad as i need to get this out the door.
Regards
Ismail
looks pretty crazy!
unless aaron has done some padding stuff in Examine, i don't think it is in there. Don't think the range fluentapi has a test written either :( will ak aaron tomorrow.
Shannon,
I have implemented ExamineEvents_GatheringNodeData and know which field aliases are range ones and they are all integers so I have done the following :
That pads them out and it seems to be working i need to get more products in there with all my range properties.
Regards
Ismail
Nice!!!! it's fixed then :)
We'll look at implementing a data type property for index fields as this will be the only way to determine the type of data going in.
I'm going to look at how best we can use https://hudson.apache.org/hudson/job/Lucene-trunk/javadoc/core/org/apache/lucene/document/NumericField.html
is .ToString("D6") still the best way of dealing with the numeric values in Lucene?
Cheers
Ali
Ali,
What is the maximum possible value for your range? if its higher that 999999 then you will need something else.
Regards
Ismail
I've got 3 properties, 2 of them between 1 to 5 and another one is can go up to 20,000,000 (Million)
what's the alternative?
Thanks
Ali
Ali,
For the big number see http://msdn.microsoft.com/en-us/library/dwhawy9k(v=vs.110).aspx#DFormatString and in c# i believe max int is 2,147,483,647 so D10 should work? Not tried it myself :-}
Regards
Ismail
Thanks Ismail, I will try it and let you know
Cheers
Ali
Hi Guys
I am desperate for help on this issue I am doing a range search and getting results back however some results shouldnt be there case and point:
Is return a results including 50000 which isnt in the range and advise on this appreciated its actually driving me mad. Its for a client could really do with getting this complete.
Andrew,
The 5000 record can you open it up in Luke and paste back how that value is stored in the index. Are using gathering node and storing a d6 value? If not then this maybe the issue.
Regards
Ismail
Thanks for coming back to sorry but can you advise how to do those things please I can't even seem to find a way install like please help.
Goto https://code.google.com/archive/p/luke/downloads download v3.5.0 you will need to install Java as well. After that double click the jar file and you can then point it to your index and browse through records in the index.
One thing did you use gathering node event to format the field you are searching over? I can see you are formatting the range values however the field in the index needs be in same format else you can get in expected results
Thank you once again so I now have luke running see screenshot attached:
See the highlighted value is included and shouldnt be, in regards gathering node event to format the field im searching how do I do this please?
I am using ez Search package https://our.umbraco.org/projects/website-utilities/ezsearch/ this installs a class ezSearchBoostrapper.cs this implements the IApplicationEventHandler interface
I have tried implementing the below is this anywhere near close?
The field you have highlighted in Luke is not d6 format. In gathering node any field you want to range search format it to d6 as you are doing when querying.
So that's the part I seem to be missing can you please advise how I format the price field to D6 then hopefully problem solved.
Andrew,
In your method GetFieldValue do
Rebuild your index. If that dont work step through your code.
Regards
Ismail
Ismail
I have tried this following:
https://our.umbraco.org/forum/using/ui-questions/16888-Examine-Search-Range-Values
The padded value appears to be getting added but I cant see it in Luke or in developer section on umbraco examine managment tab.
ExamineIndex.config
ExamineSettings.config:
Did you rebuild the index? Also have you tried stepping through to see if that code gets hit?
Regards
Ismail
Yes on both counts rebuilt the index using Umbraco developer section and stepped through and I can see __paddedPrice getting added and formatted correctly but unfortunately still not appaearing in luke or index.
Seem to be getting added to the content field but im expecting another field __paddedPrice
Regards
Andrew
OMG ive got it FINALLY i can sleep at night once more:
Removing:
And Replacing with:
It then appeared in luke and in the index in umbraco and I am now getting the expected results when querying in Luke:
I cant thank you enough for your help massive respect. Now just to test in the app.
Hmm weird but glad you fixed it.
Btw there is one space left on https://umbraco.com/training/book-courses/searching-and-indexing/ if your up for it lol
is working on a reply...