Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Aug 18, 2010 @ 13:30
    Ismail Mayat
    0

    Examine range query numeric

    I have a range query it spits out lucene query:

    +(+nodeTypeAlias:product +__budgetFrom:[0 TO 101]) +__IndexType:content

    when i run the code i get no results. When i put the query through luke i get 2 results. In my index i have 3 product items with budgetFrom (in index property is in as __budgetFrom) the values are 50,100,100 for that query i would expect it to return 3 items both in luke and via code.

    Is it treating that numeric value as string? see http://wiki.apache.org/lucene-java/SearchNumericalFields

    Regards

    Ismail

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Aug 18, 2010 @ 13:35
    Ismail Mayat
    0

    Right in answer to my question it does treat as string you need to pad out when adding to index see http://stackoverflow.com/questions/708075/problem-using-lucene-rangequery


  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Aug 18, 2010 @ 13:45
    Ismail Mayat
    1

    Ok another update i now can get results same as luke i changed my code from

    IBooleanOperation Range(string fieldName, int start, int end, bool includeLower, bool includeUpper);

    to

    IBooleanOperation Range(string fieldName, string start, string end);

    So just need to do the padding thing and should be onto a winner!

  • Alex Norcliffe 222 posts 287 karma points
    Aug 18, 2010 @ 14:15
    Alex Norcliffe
    1

    Hi Ismail, Lucene indexers should provide either the padding or the field type for you based on the incoming data. Plus, more recent version have a numeric field type which avoids you having to do this kind of padding.

    Having chatted with Shannon before about his implementation for Examine, I'm pretty sure he's doing this somewhere. I'll punt this thread Shannon's way and hopefully he can shed some light on it for you, to avoid doing the padding yourself.

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Aug 18, 2010 @ 14:23
    Ismail Mayat
    0

    Alex,

    Awesome. Given the fact we have IBooleanOperation Range(string fieldName, int start, int end, bool includeLower, bool includeUpper);  I would assume that its being handled as well.

    Regards

    Ismail

  • Aaron Powell 1708 posts 3044 karma points c-trib
    Aug 18, 2010 @ 14:33
    Aaron Powell
    0

    Yeah internally we're using NumericRangeQuery to build it up. 

    What may be a problem is we're not pushing the data into Lucene using any numeric tools. I started looking at this the other day but I got a bit confused (and so far we hadn't had any problem with the current method) and left it alone.

    I am getting a Lucene book shortly so I'll have another look once I get a better understanding of how to handle numbers with Lucene ;)

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Aug 18, 2010 @ 14:41
    Ismail Mayat
    0

    I need to order my copy of lucene in action v2 awesome book used to have v1 at my last work place. For now im gonna pad as i need to get this out the door.

    Regards

    Ismail

  • Shannon Deminick 1519 posts 5237 karma points
    Aug 18, 2010 @ 16:47
    Shannon Deminick
    0

    looks pretty crazy!

    unless aaron has done some padding stuff in Examine, i don't think it is in there. Don't think the range fluentapi has a test written either :( will ak aaron tomorrow.

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Aug 18, 2010 @ 17:03
    Ismail Mayat
    0

    Shannon,

    I have implemented ExamineEvents_GatheringNodeData and know which field aliases are range ones and they are all integers so I have done the following :

    e.Fields.Add("__" + luceneFieldAlias, i.ToString("D6"));

    That pads them out and it seems to be working i need to get more products in there with all my range properties.

    Regards

    Ismail   

  • Shannon Deminick 1519 posts 5237 karma points
    Aug 19, 2010 @ 01:39
    Shannon Deminick
    0

    Nice!!!! it's fixed then :)

    We'll look at implementing a data type property for index fields as this will be the only way to determine the type of data going in.

  • Aaron Powell 1708 posts 3044 karma points c-trib
    Aug 19, 2010 @ 03:06
  • Ali Sheikh Taheri 470 posts 1647 karma points c-trib
    Dec 01, 2013 @ 14:52
    Ali Sheikh Taheri
    0

    is .ToString("D6") still the best way of dealing with the numeric values in Lucene?

    Cheers

    Ali

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Dec 02, 2013 @ 14:32
    Ismail Mayat
    0

    Ali,

    What is the maximum possible value for your range? if its higher that 999999 then you will need something else.

    Regards

    Ismail

  • Ali Sheikh Taheri 470 posts 1647 karma points c-trib
    Dec 02, 2013 @ 14:48
    Ali Sheikh Taheri
    0

    I've got 3 properties, 2 of them between 1 to 5 and another one is can go up to 20,000,000 (Million)

    what's the alternative?

    Thanks

    Ali

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Dec 02, 2013 @ 15:30
    Ismail Mayat
    0

    Ali,

    For the big number see http://msdn.microsoft.com/en-us/library/dwhawy9k(v=vs.110).aspx#DFormatString and in c# i believe max int is 2,147,483,647 so D10 should work? Not tried it myself :-}

    Regards

    Ismail

  • Ali Sheikh Taheri 470 posts 1647 karma points c-trib
    Dec 02, 2013 @ 16:14
    Ali Sheikh Taheri
    0

    Thanks Ismail, I will try it and let you know

    Cheers

    Ali

  • Andrew Bright 67 posts 217 karma points
    Mar 11, 2017 @ 00:10
    Andrew Bright
    0

    Hi Guys

    I am desperate for help on this issue I am doing a range search and getting results back however some results shouldnt be there case and point:

    int Model.MinPrice = 300000
    int Model.MaxPrice = 500000
    
     query.And().Range("price", model.MinPrice.ToString("D6"), model.MaxPrice.ToString("D6"));
    

    Is return a results including 50000 which isnt in the range and advise on this appreciated its actually driving me mad. Its for a client could really do with getting this complete.

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Mar 11, 2017 @ 10:40
    Ismail Mayat
    1

    Andrew,

    The 5000 record can you open it up in Luke and paste back how that value is stored in the index. Are using gathering node and storing a d6 value? If not then this maybe the issue.

    Regards

    Ismail

  • Andrew Bright 67 posts 217 karma points
    Mar 11, 2017 @ 11:25
    Andrew Bright
    0

    Thanks for coming back to sorry but can you advise how to do those things please I can't even seem to find a way install like please help.

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Mar 11, 2017 @ 11:37
    Ismail Mayat
    1

    Goto https://code.google.com/archive/p/luke/downloads download v3.5.0 you will need to install Java as well. After that double click the jar file and you can then point it to your index and browse through records in the index.

    One thing did you use gathering node event to format the field you are searching over? I can see you are formatting the range values however the field in the index needs be in same format else you can get in expected results

  • Andrew Bright 67 posts 217 karma points
    Mar 11, 2017 @ 22:22
    Andrew Bright
    0

    Thank you once again so I now have luke running see screenshot attached: enter image description here

    See the highlighted value is included and shouldnt be, in regards gathering node event to format the field im searching how do I do this please?

    I am using ez Search package https://our.umbraco.org/projects/website-utilities/ezsearch/ this installs a class ezSearchBoostrapper.cs this implements the IApplicationEventHandler interface

    #region Application Event Handlers
    
    public void OnApplicationInitialized(UmbracoApplicationBase umbracoApplication, 
        ApplicationContext applicationContext)
    { }
    
    public void OnApplicationStarting(UmbracoApplicationBase umbracoApplication, 
        ApplicationContext applicationContext)
    { }
    
    public void OnApplicationStarted(UmbracoApplicationBase umbracoApplication, 
        ApplicationContext applicationContext)
    {
        ExamineManager.Instance.IndexProviderCollection["ExternalIndexer"]
            .GatheringNodeData += OnGatheringNodeData;
    }
    

    I have tried implementing the below is this anywhere near close?

    private void OnGatheringNodeData(object sender, IndexingNodeDataEventArgs e)
    {
    
        Node theNode = new Node(e.NodeId);
        //check if this is 'Content' (as opposed to media, etc...)
        if (e.IndexType == IndexTypes.Content)
        {
            if (theNode.NodeTypeAlias == "umbPropertyDetails")
            {
                var node = new Node(e.NodeId);
    
                var propertyValue = TpayneandCo.App_Code.ExamineEventsHelper.GetPropertyValue(theNode, "price");
    
                if (propertyValue != string.Empty)
                {
    
                    e.Fields.Add("__" + "price", GetFieldValue(e, propertyValue, "price"));
    
                }
    
            }
    
            AddToContentsField(e);
    
        }
    
    }
    
    private void AddToContentsField(IndexingNodeDataEventArgs e)
    {
    
        Dictionary<string, string> fields = e.Fields;
    
        var combinedFields = new StringBuilder();
    
        foreach (KeyValuePair<string, string> keyValuePair in fields)
        {
    
            combinedFields.AppendLine(keyValuePair.Value);
    
        }
    
        e.Fields.Add("contents", combinedFields.ToString());
    
    }
    
    private string GetFieldValue(IndexingNodeDataEventArgs e, string propertyValue, string luceneFieldAlias)
    {
    
        int nodeId = 0;
    
        int.TryParse(propertyValue, out nodeId);
    
        var n = new Node(nodeId);
    
        //node does not exist but we have numeric value
    
        if (n.Id != 0)
        {
    
            //have to pad out to get lucene range queries to work
    
            int i = 0;
    
            int.TryParse(n.Name, out i);
    
            return i.ToString("00000000");
    
        }
    
        return nodeId.ToString("00000000");
    
    }
    
  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Mar 12, 2017 @ 09:14
    Ismail Mayat
    1

    The field you have highlighted in Luke is not d6 format. In gathering node any field you want to range search format it to d6 as you are doing when querying.

  • Andrew Bright 67 posts 217 karma points
    Mar 12, 2017 @ 11:55
    Andrew Bright
    0

    So that's the part I seem to be missing can you please advise how I format the price field to D6 then hopefully problem solved.

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Mar 13, 2017 @ 08:14
    Ismail Mayat
    1

    Andrew,

    In your method GetFieldValue do

    i.ToString("D6");
    

    Rebuild your index. If that dont work step through your code.

    Regards

    Ismail

  • Andrew Bright 67 posts 217 karma points
    Mar 13, 2017 @ 13:17
    Andrew Bright
    0

    Ismail

    I have tried this following:

    https://our.umbraco.org/forum/using/ui-questions/16888-Examine-Search-Range-Values

    The padded value appears to be getting added but I cant see it in Luke or in developer section on umbraco examine managment tab.

    public class ExamineEvents : ApplicationBase
    {
        public ExamineEvents()
        {
            //Add event handler for 'GatheringNodeData' 
            ExamineManager.Instance.IndexProviderCollection["ExternalIndexer"].GatheringNodeData += ExamineEvents_GatheringNodeData;
    
        }
    
        void ExamineEvents_GatheringNodeData(object sender, IndexingNodeDataEventArgs e)
        {
            Node theNode = new Node(e.NodeId);
            //check if this is 'Content' (as opposed to media, etc...)
            if (e.IndexType == IndexTypes.Content)
            {
    
                if (theNode.NodeTypeAlias == "Patient")
                {
                    //var node = new Node(e.NodeId);
                    var propertyValue = ExamineEventsHelper.GetPropertyValue(theNode, "price");
    
                    if (propertyValue != string.Empty)
                    {
                        e.Fields.Add("__" + "paddedPrice", GetFieldValue(e, propertyValue, "price"));
                    }
    
                }
    
                AddToContentsField(e);
    
            }
    
        }
    
        private void AddToContentsField(IndexingNodeDataEventArgs e)
        {
    
            Dictionary<string, string> fields = e.Fields;
    
            var combinedFields = new StringBuilder();
    
            foreach (KeyValuePair<string, string> keyValuePair in fields)
            {
    
                combinedFields.AppendLine(keyValuePair.Value);
    
            }
    
            e.Fields.Add("contents", combinedFields.ToString());
    
        }
    
        private string GetFieldValue(IndexingNodeDataEventArgs e, string propertyValue, string luceneFieldAlias)
        {
    
            int nodeId = e.NodeId;
    
            //int.TryParse(propertyValue, out nodeId);
    
            var n = new Node(nodeId);
    
            //node does not exist but we have numeric value
            if (n.Id != 0)
            {
                //have to pad out to get lucene range queries to work
                int i = 0;
    
                //int.TryParse(n.Name, out i);
                int.TryParse(propertyValue, out i);
    
                return i.ToString("D6");
            }
            return nodeId.ToString("D6");
        }
    
    }
    

    ExamineIndex.config

    <?xml version="1.0"?>
    <!-- 
    Umbraco examine is an extensible indexer and search engine.
    This configuration file can be extended to create your own index sets.
    Index/Search providers can be defined in the UmbracoSettings.config
    
    More information and documentation can be found on CodePlex: http://umbracoexamine.codeplex.com
    -->
    <ExamineLuceneIndexSets>
      <!-- The internal index set used by Umbraco back-office - DO NOT REMOVE -->
      <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/Internal/"/>
    
      <!-- The internal index set used by Umbraco back-office for indexing members - DO NOT REMOVE -->
      <IndexSet SetName="InternalMemberIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/InternalMember/">
        <IndexAttributeFields>
          <add Name="id" />
          <add Name="nodeName"/>
          <add Name="updateDate" />
          <add Name="writerName" />
          <add Name="loginName" />
          <add Name="email" />
          <add Name="nodeTypeAlias" />
        </IndexAttributeFields>
      </IndexSet>
    
      <!-- Default Indexset for external searches, this indexes all fields on all types of nodes-->
      <IndexSet SetName="ExternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/External/">
        <IndexAttributeFields>
          <add Name="id" />
          <add Name="nodeName"/>
          <add Name="nodeTypeAlias" />
          <add Name="parentID" />
        </IndexAttributeFields>
        <IndexUserFields/>
        <IncludeNodeTypes/>
        <ExcludeNodeTypes />
      </IndexSet>
    </ExamineLuceneIndexSets>
    

    ExamineSettings.config:

    <?xml version="1.0"?>
    <!-- 
    Umbraco examine is an extensible indexer and search engine.
    This configuration file can be extended to add your own search/index providers.
    Index sets can be defined in the ExamineIndex.config if you're using the standard provider model.
    
    More information and documentation can be found on CodePlex: http://umbracoexamine.codeplex.com
    -->
    <Examine>
      <ExamineIndexProviders>
        <providers>
          <add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
               supportUnpublished="true"
               supportProtected="true"
               analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>
    
          <add name="InternalMemberIndexer" type="UmbracoExamine.UmbracoMemberIndexer, UmbracoExamine"
               supportUnpublished="true"
               supportProtected="true"
               analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"/>
    
            <!-- default external indexer, which excludes protected and unpublished pages-->
          <add name="ExternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
              runAsync="true"
              supportUnpublished="false"
              supportProtected="true"
              interval="10"
              analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
              indexSet="ExternalIndexSet"/>
    
        </providers>
      </ExamineIndexProviders>
    
      <ExamineSearchProviders defaultProvider="ExternalSearcher">
        <providers>
          <add name="InternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
               analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>
    
          <add name="ExternalSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
              analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
              indexSet="ExternalIndexSet"/>
    
          <add name="InternalMemberSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
               analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net" enableLeadingWildcard="true"/>
    
        </providers>
      </ExamineSearchProviders>
    
    </Examine>
    
  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Mar 13, 2017 @ 13:35
    Ismail Mayat
    1

    Did you rebuild the index? Also have you tried stepping through to see if that code gets hit?

    Regards

    Ismail

  • Andrew Bright 67 posts 217 karma points
    Mar 13, 2017 @ 14:30
    Andrew Bright
    0

    Yes on both counts rebuilt the index using Umbraco developer section and stepped through and I can see __paddedPrice getting added and formatted correctly but unfortunately still not appaearing in luke or index.

    Seem to be getting added to the content field but im expecting another field __paddedPrice

    Regards

    Andrewenter image description here

  • Andrew Bright 67 posts 217 karma points
    Mar 13, 2017 @ 15:13
    Andrew Bright
    0

    OMG ive got it FINALLY i can sleep at night once more:

    Removing:

    e.Fields.Add("__" + "paddedPrice", GetFieldValue(e, propertyValue, "price"));
    

    And Replacing with:

    e.Fields.Add("paddedPrice", GetFieldValue(e, propertyValue, "price"));
    

    It then appeared in luke and in the index in umbraco and I am now getting the expected results when querying in Luke:

    enter image description here

    I cant thank you enough for your help massive respect. Now just to test in the app.

  • Ismail Mayat 4511 posts 10061 karma points MVP 2x admin c-trib
    Mar 13, 2017 @ 16:14
    Ismail Mayat
    0

    Hmm weird but glad you fixed it.

    Btw there is one space left on https://umbraco.com/training/book-courses/searching-and-indexing/ if your up for it lol

Please Sign in or register to post replies

Write your reply to:

Draft