Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Yuri Derevianko 3 posts 74 karma points
    Jul 30, 2024 @ 19:37
    Yuri Derevianko
    0

    ExamineX custom index

    Hello community,

    I'm trying to create custom index with ExamineX using Azure AI search service on empty Umbraco 10. Currently I use free tier and no license for ExamineX, I disabled MembersIndex in settings and just want to create custom NewsIndex that will store only nodes of content type 'news' — this is going to be 3rd index available in such setup.

    As a roadmap I use this page https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/reference/searching/examine/indexing about how to create custom index with local Examine. Also I read quite thoroughly https://examinex.online. I have partial success which is my NewsIndex is created in Azure Service and seen in Umbraco. However I can not properly implement publish/rebuild logic for it — whatever I do, it throws NRE somewhere in ExamineX.Shared.Umbraco code with no info I can use.

    Just wander is there may be any kind of step-by-step instructions for this? Could not find it. Or will appreciate some code snippet :) Thanks in advance.

  • Lewis Smith 211 posts 620 karma points c-trib
    Sep 19, 2024 @ 11:19
    Lewis Smith
    0

    Hi Yuri,

    Im having the same issue and finding myself going round in circles.

    Did you get anywhere towards resolving this?

    Lewis

  • Lewis Smith 211 posts 620 karma points c-trib
    Sep 19, 2024 @ 13:34
    Lewis Smith
    0

    So i got to the bottom of this one. (Or my colleague did)

    Here is my custom index class, as well as the but that needs adding to the composer.

    public class MyCustomIndex : UmbracoContentAzureSearchIndex
    {
        public MyCustomIndex (
            ILoggerFactory loggerFactory,
            string name,
            LicenseManagerCollection licenseManager,
            IOptions<ExamineXConfig> config,
            IOptionsMonitor<AzureSearchIndexOptions> indexOptions,
            IOptions<AzureSearchOptions> azureSearchOptions,
            IServerRoleAccessor serverRole)
            : base(loggerFactory, name, licenseManager, config, indexOptions, azureSearchOptions, serverRole)
        {
        }
    }
    
    builder.Services.AddExamineXAzureSearchIndexFromUmbraco<AzureSearchFilterIndex>("MyCustomIndex");
    

    So going from normal Examine (disc) to ExamineX, you can update your custom index to inherit UmbracoContentAzureSearchIndex and update your composer to use AddExamineXAzureSearchIndexFromUmbraco.

  • Yuri Derevianko 3 posts 74 karma points
    Sep 23, 2024 @ 08:45
    Yuri Derevianko
    0

    Hi Lewis, sorry, missed your previous comment. Thanks for the info re index class, actually a week ago I came to that by myself also. Now I'm trying to get the index properly populated. I intentionally added extra logging and see that content nodes are indeed get indexed. But by some reason in backend Umbraco still does not show expected status.

    My classes are quite close to Umbraco docs. Here is Populator method:

        protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes)
    {
        foreach (IIndex index in indexes)
        {
            //Go over all the content and index it if it fits the criteria
            IContent[] roots = _contentService.GetRootContent().ToArray();
            index.IndexItems(_newsIndexValueSetBuilder.GetValueSets(roots));
    
            foreach (var root in roots)
            {
                const int pageSize = 10000;
                var pageIndex = 0;
                IContent[] descendants;
                //Weird syntax, but it follows the Umbraco docs
                do
                {
                    descendants = _contentService.GetPagedDescendants(root.Id, pageIndex, pageSize, out _).ToArray();
                    IEnumerable<ValueSet> valueSets = _newsIndexValueSetBuilder.GetValueSets(descendants);
                    index.IndexItems(valueSets);
                }
                while (descendants.Length == pageSize);
            }
        }
    }
    

    and here is GetValueSets method with extra logging. Here I am adding all the content node's properties as index field:

        public IEnumerable<ValueSet> GetValueSets(params IContent[] contents)
    {
        foreach (var content in contents.Where(x => _contentTypes.Contains(x.ContentType.Alias) && x.Published))
        {
            _logger.LogDebug("---------------- Indexing content item ID: {id}, name: {name}", content.Id, content.Name);
    
            var indexValues = new Dictionary<string, object>
            {
                //This is where we assign our fields defined in options
                [UmbracoExamineFieldNames.NodeNameFieldName] = content.Name!,
                ["id"] = content.Id,
                ["parentId"] = content.ParentId,
                ["nodeName"] = content.Name!,
                [SearchKeys.POSTED_DATE_FIELD] = content.CreateDate,
                [SearchKeys.POSTED_DATE_TICKS_FIELD] = content.CreateDate.Ticks
            };
    
            foreach (var property in content.Properties) indexValues[property.Alias] = property.GetValue();
    
            var valueSet = new ValueSet(content.Id.ToString(), IndexTypes.Content, content.ContentType.Alias, indexValues);
    
            yield return valueSet;
        }
    }
    

    Log messages from this method come through and I see them in Umbraco logs, but index is still empty and configured for 8 fields instead of many more. In Azure Search service - the same....... So still fighting..

    enter image description here

  • Yuri Derevianko 3 posts 74 karma points
    Sep 28, 2024 @ 17:49
    Yuri Derevianko
    1

    In case anyone is interested in the topic, here is very basic Umbraco solution with custom ExamineX index configuration - github

  • Johan Runsten 41 posts 280 karma points c-trib
    22 days ago
    Johan Runsten
    0

    Thank you, this helped! I'm evaluating ExamineX and my 100 allowed items were taken up by media items so I was able to exclude them from the ExternalIndex pretty easily.

  • Shannon Deminick 1530 posts 5278 karma points MVP 3x
    1 week ago
    Shannon Deminick
    1

    Hi all - We will update the documentation for custom indexes today.

    Its actually easier than what is listed above and you don't need to inherit from UmbracoContentAzureSearchIndex.

    For custom indexes with Examine, you can continue to inherit from the normal/documented index types based on the Umbraco docs https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/reference/searching/examine/indexing

    Then, to opt your custom index into ExamineX (probably based on your environment type like 'prod'), you can just do:

    • For Umbraco Content based indexes (i.e. UmbracoContentIndex/IUmbracoContentIndex):

      • builder.Services .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>("YourIndexName")
    • For non-Umbraco custom indexes based on custom data sources (i.e. UmbracoExamineIndex/IUmbracoIndex):

      • builder.Services .AddExamineXAzureSearchIndex<AzureSearchIndex>("YourIndexName")
  • Shannon Deminick 1530 posts 5278 karma points MVP 3x
    1 week ago
    Shannon Deminick
    0
  • Sue 3 posts 73 karma points notactivated
    4 days ago
    Sue
    0

    I am really struggling to get this to work. I followed the umbraco code to the letter and have it working perfectly with local indexes.

    I have a custom index which only needs a few custom properties based on the content being indexed (such as converting a location to lat/long for example).

    However when I try to switch it to an ExamineX index (which are working for Internal/External indexes) I get a load of different issues:

    • object reference not set errors when running the IndexItems code in the populator
    • sometimes index simply does not populate with no errors being logged
    • when publishing content, the custom fields are not being indexed, instead the umbraco properties

    Can I confirm whether the existing line of config should be removed....or should both lines exist? The documentation could be interpreted either way.

    enter image description here

    Should any other code changes be required to swap from a local to azure index using ExamineX? A full code example would be very useful, that works both when building an index and publishing content with custom fields being added in the valuesetbuilder.

  • Shannon Deminick 1530 posts 5278 karma points MVP 3x
    3 days ago
    Shannon Deminick
    1

    The ExamineX extension methods should not be done within a composer.

    Ordering is important and composers execute randomly within the call to

    services.AddUmbraco(...)...AddComposers()

    The ExamineX call to AddExamineXAzureSearch() on the Umbraco builder since ExamineX version 6 was done for a reason - because composers are difficult to manage if there are cross dependencies between them and ordering is important.

    As an example, your startup code should look something like

            services.AddUmbraco(_env, _config)
                .AddBackOffice()
                .AddWebsite()
                .AddDeliveryApi()
                .AddComposers()
                .AddExamineXAzureSearch()
                .Build();
    

    Within AddExamineXAzureSearch(), it does this:

            // register the ExamineX indexes which will wrap the options from the original umbraco/user options
            builder.Services
                .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>(Constants.UmbracoIndexes.InternalIndexName)
                .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>(Constants.UmbracoIndexes.ExternalIndexName)
                .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchMemberIndex>(Constants.UmbracoIndexes.MembersIndexName)
                .AddExamineXAzureSearchIndexFromUmbraco<UmbracoDeliveryApiContentIndex>(Constants.UmbracoIndexes.DeliveryApiContentIndexName)
                .ConfigureOptions<ConfigureIndexOptions>();
    

    So if you are using the AddExamineXAzureSearchIndexFromUmbraco for a custom lucene based index, call this after the call to AddExamineXAzureSearch

    I can look into adding a full code example to the ExamineX docs.

  • Sue 3 posts 73 karma points notactivated
    1 day ago
    Sue
    0

    Hi Shannon, I will try moving these round however a full code sample would be much appreciated!

    Just to create a basic custom index (with nodeid, alias and including a few derived fields, does not need the full set of umbraco properties) and to configure this with examineX so it works reliably when doing an index rebuild and publishing content.

    Thanks

  • Johan Runsten 41 posts 280 karma points c-trib
    4 days ago
    Johan Runsten
    0

    You should remove the AddExamineLuceneIndex call and only leave the ExamineX call. There are 2 different ExamineX calls depending on whether your custom index is of type UmbracoContentIndex/IUmbracoContentIndex (AddExamineXAzureSearchIndexFromUmbraco) or UmbracoExamineIndex/IUmbracoIndex (AddExamineXAzureSearchIndex).

  • Sue 3 posts 73 karma points notactivated
    4 days ago
    Sue
    0

    Thanks, that's what I'd originally tried but just get the errors/issues mentioned.

    I'm assuming it's the 'umbraco' one I need but it seems to just use the full set of umbraco content properties when publishing a node.

    Using the 'custom' one breaks the back office Examine dashboard.

  • Johan Runsten 41 posts 280 karma points c-trib
    3 days ago
    Johan Runsten
    0

    Hmm OK sorry I don't have any experience with UmbracoContentIndex, only the UmbracoExamineIndex and then adding my values from scratch.

    I am however going down the same path as you (creating a custom index based on a content index but with extra values), but I haven't gotten to it yet :)

  • RM 36 posts 98 karma points
    1 day ago
    RM
    0

    @Shannon I am helping Sue work through this issue, not sure if its just the application in general but i stripped the index creation back as we have spoken before about the setup. As we have also discussed, if its not working on Azure then stripping it back to the local indexes is a good place to start, which i have done but having a very odd issue.

    We use the setup below in a couple of different Umbraco installs and its fine, so its very odd, but we are not getting any feedback from Examine / Umbraco to understand if anything is failing as if it is, its doing it silently.

    Basically we have:

    • A ConfigureWebsiteIndexOptions class which inherits IConfigureNamedOptions as we have some custom field definitions
    • A WebsiteIndexerComponent class that Initializes the TransformingIndexValues method and a composer to register it

    Startup has this code:

            services
            .AddExamineLuceneIndex<UmbracoContentIndex, ConfigurationEnabledDirectoryFactory>(SearchKeys.WEBSITE_INDEXER)
            .ConfigureOptions<ConfigureWebsiteIndexOptions>();
    

    The WebsiteIndexOptions Gets fired, the WebsiteIndexerComponent gets fired and i can see it registering the event handler but whilst the index is created, it has no records and no fields.

    If we try and publish the index, nothing happens (we had the same issue with ExamineX Azure for this setup, no logging apart from the initial index rebuild notification).

    But, if i publish a record with a document type that is allowed to be indexed by this custom index, it actually hits the TransformingIndexValues method and adds that record to the index.

    Its like the index is being provisioned but not correctly wired up to anything, but the system knows about it because publishing an item pushes that item into the index. We have had issues before with nothing firing, but i don't think we have ever had an issue where it only fires for publishing. I may attempt to create a blank Umbraco project and point to the database and just migrate the index code, but is there any specific reason why this could be the case?

  • Shannon Deminick 1530 posts 5278 karma points MVP 3x
    1 day ago
    Shannon Deminick
    0

    Yeah I'd suggest maybe starting with a new site and keep things as simple as possible. Depending on what tier you are using, you also might be hitting Azure Search limitations. If these limits are breached in any way (even in the payload), the indexing fails silently unfortunately. https://github.com/SDKits/ExamineX/issues/5

    More than happy to assist everyone to ensure you get this working. FYI I'm away next week so if I don't reply that is why.

    Might also be worth asking on the ExamineX GitHub support since we don't normally monitor this forum that closely.

Please Sign in or register to post replies

Write your reply to:

Draft