I'm trying to create custom index with ExamineX using Azure AI search service on empty Umbraco 10. Currently I use free tier and no license for ExamineX, I disabled MembersIndex in settings and just want to create custom NewsIndex that will store only nodes of content type 'news' — this is going to be 3rd index available in such setup.
As a roadmap I use this page https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/reference/searching/examine/indexing about how to create custom index with local Examine. Also I read quite thoroughly https://examinex.online. I have partial success which is my NewsIndex is created in Azure Service and seen in Umbraco. However I can not properly implement publish/rebuild logic for it — whatever I do, it throws NRE somewhere in ExamineX.Shared.Umbraco code with no info I can use.
Just wander is there may be any kind of step-by-step instructions for this? Could not find it. Or will appreciate some code snippet :) Thanks in advance.
So going from normal Examine (disc) to ExamineX, you can update your custom index to inherit UmbracoContentAzureSearchIndex and update your composer to use AddExamineXAzureSearchIndexFromUmbraco.
Hi Lewis, sorry, missed your previous comment. Thanks for the info re index class, actually a week ago I came to that by myself also. Now I'm trying to get the index properly populated. I intentionally added extra logging and see that content nodes are indeed get indexed. But by some reason in backend Umbraco still does not show expected status.
My classes are quite close to Umbraco docs. Here is Populator method:
protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes)
{
foreach (IIndex index in indexes)
{
//Go over all the content and index it if it fits the criteria
IContent[] roots = _contentService.GetRootContent().ToArray();
index.IndexItems(_newsIndexValueSetBuilder.GetValueSets(roots));
foreach (var root in roots)
{
const int pageSize = 10000;
var pageIndex = 0;
IContent[] descendants;
//Weird syntax, but it follows the Umbraco docs
do
{
descendants = _contentService.GetPagedDescendants(root.Id, pageIndex, pageSize, out _).ToArray();
IEnumerable<ValueSet> valueSets = _newsIndexValueSetBuilder.GetValueSets(descendants);
index.IndexItems(valueSets);
}
while (descendants.Length == pageSize);
}
}
}
and here is GetValueSets method with extra logging. Here I am adding all the content node's properties as index field:
public IEnumerable<ValueSet> GetValueSets(params IContent[] contents)
{
foreach (var content in contents.Where(x => _contentTypes.Contains(x.ContentType.Alias) && x.Published))
{
_logger.LogDebug("---------------- Indexing content item ID: {id}, name: {name}", content.Id, content.Name);
var indexValues = new Dictionary<string, object>
{
//This is where we assign our fields defined in options
[UmbracoExamineFieldNames.NodeNameFieldName] = content.Name!,
["id"] = content.Id,
["parentId"] = content.ParentId,
["nodeName"] = content.Name!,
[SearchKeys.POSTED_DATE_FIELD] = content.CreateDate,
[SearchKeys.POSTED_DATE_TICKS_FIELD] = content.CreateDate.Ticks
};
foreach (var property in content.Properties) indexValues[property.Alias] = property.GetValue();
var valueSet = new ValueSet(content.Id.ToString(), IndexTypes.Content, content.ContentType.Alias, indexValues);
yield return valueSet;
}
}
Log messages from this method come through and I see them in Umbraco logs, but index is still empty and configured for 8 fields instead of many more. In Azure Search service - the same....... So still fighting..
Thank you, this helped! I'm evaluating ExamineX and my 100 allowed items were taken up by media items so I was able to exclude them from the ExternalIndex pretty easily.
I am really struggling to get this to work. I followed the umbraco code to the letter and have it working perfectly with local indexes.
I have a custom index which only needs a few custom properties based on the content being indexed (such as converting a location to lat/long for example).
However when I try to switch it to an ExamineX index (which are working for Internal/External indexes) I get a load of different issues:
object reference not set errors when running the IndexItems code in the populator
sometimes index simply does not populate with no errors being logged
when publishing content, the custom fields are not being indexed, instead the umbraco properties
Can I confirm whether the existing line of config should be removed....or should both lines exist? The documentation could be interpreted either way.
Should any other code changes be required to swap from a local to azure index using ExamineX?
A full code example would be very useful, that works both when building an index and publishing content with custom fields being added in the valuesetbuilder.
The ExamineX extension methods should not be done within a composer.
Ordering is important and composers execute randomly within the call to
services.AddUmbraco(...)...AddComposers()
The ExamineX call to AddExamineXAzureSearch() on the Umbraco builder since ExamineX version 6 was done for a reason - because composers are difficult to manage if there are cross dependencies between them and ordering is important.
As an example, your startup code should look something like
// register the ExamineX indexes which will wrap the options from the original umbraco/user options
builder.Services
.AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>(Constants.UmbracoIndexes.InternalIndexName)
.AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>(Constants.UmbracoIndexes.ExternalIndexName)
.AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchMemberIndex>(Constants.UmbracoIndexes.MembersIndexName)
.AddExamineXAzureSearchIndexFromUmbraco<UmbracoDeliveryApiContentIndex>(Constants.UmbracoIndexes.DeliveryApiContentIndexName)
.ConfigureOptions<ConfigureIndexOptions>();
So if you are using the AddExamineXAzureSearchIndexFromUmbraco for a custom lucene based index, call this after the call to AddExamineXAzureSearch
I can look into adding a full code example to the ExamineX docs.
Hi Shannon,
I will try moving these round however a full code sample would be much appreciated!
Just to create a basic custom index (with nodeid, alias and including a few derived fields, does not need the full set of umbraco properties) and to configure this with examineX so it works reliably when doing an index rebuild and publishing content.
You should remove the AddExamineLuceneIndex call and only leave the ExamineX call. There are 2 different ExamineX calls depending on whether your custom index is of type UmbracoContentIndex/IUmbracoContentIndex (AddExamineXAzureSearchIndexFromUmbraco) or UmbracoExamineIndex/IUmbracoIndex (AddExamineXAzureSearchIndex).
Hmm OK sorry I don't have any experience with UmbracoContentIndex, only the UmbracoExamineIndex and then adding my values from scratch.
I am however going down the same path as you (creating a custom index based on a content index but with extra values), but I haven't gotten to it yet :)
@Shannon I am helping Sue work through this issue, not sure if its just the application in general but i stripped the index creation back as we have spoken before about the setup. As we have also discussed, if its not working on Azure then stripping it back to the local indexes is a good place to start, which i have done but having a very odd issue.
We use the setup below in a couple of different Umbraco installs and its fine, so its very odd, but we are not getting any feedback from Examine / Umbraco to understand if anything is failing as if it is, its doing it silently.
Basically we have:
A ConfigureWebsiteIndexOptions class which inherits IConfigureNamedOptions as we have some custom field definitions
A WebsiteIndexerComponent class that Initializes the TransformingIndexValues method and a composer to register it
The WebsiteIndexOptions Gets fired, the WebsiteIndexerComponent gets fired and i can see it registering the event handler but whilst the index is created, it has no records and no fields.
If we try and publish the index, nothing happens (we had the same issue with ExamineX Azure for this setup, no logging apart from the initial index rebuild notification).
But, if i publish a record with a document type that is allowed to be indexed by this custom index, it actually hits the TransformingIndexValues method and adds that record to the index.
Its like the index is being provisioned but not correctly wired up to anything, but the system knows about it because publishing an item pushes that item into the index. We have had issues before with nothing firing, but i don't think we have ever had an issue where it only fires for publishing. I may attempt to create a blank Umbraco project and point to the database and just migrate the index code, but is there any specific reason why this could be the case?
Yeah I'd suggest maybe starting with a new site and keep things as simple as possible. Depending on what tier you are using, you also might be hitting Azure Search limitations. If these limits are breached in any way (even in the payload), the indexing fails silently unfortunately. https://github.com/SDKits/ExamineX/issues/5
More than happy to assist everyone to ensure you get this working. FYI I'm away next week so if I don't reply that is why.
Might also be worth asking on the ExamineX GitHub support since we don't normally monitor this forum that closely.
ExamineX custom index
Hello community,
I'm trying to create custom index with ExamineX using Azure AI search service on empty Umbraco 10. Currently I use free tier and no license for ExamineX, I disabled MembersIndex in settings and just want to create custom NewsIndex that will store only nodes of content type 'news' — this is going to be 3rd index available in such setup.
As a roadmap I use this page https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/reference/searching/examine/indexing about how to create custom index with local Examine. Also I read quite thoroughly https://examinex.online. I have partial success which is my NewsIndex is created in Azure Service and seen in Umbraco. However I can not properly implement publish/rebuild logic for it — whatever I do, it throws NRE somewhere in ExamineX.Shared.Umbraco code with no info I can use.
Just wander is there may be any kind of step-by-step instructions for this? Could not find it. Or will appreciate some code snippet :) Thanks in advance.
Hi Yuri,
Im having the same issue and finding myself going round in circles.
Did you get anywhere towards resolving this?
Lewis
So i got to the bottom of this one. (Or my colleague did)
Here is my custom index class, as well as the but that needs adding to the composer.
So going from normal Examine (disc) to ExamineX, you can update your custom index to inherit
UmbracoContentAzureSearchIndex
and update your composer to useAddExamineXAzureSearchIndexFromUmbraco
.Hi Lewis, sorry, missed your previous comment. Thanks for the info re index class, actually a week ago I came to that by myself also. Now I'm trying to get the index properly populated. I intentionally added extra logging and see that content nodes are indeed get indexed. But by some reason in backend Umbraco still does not show expected status.
My classes are quite close to Umbraco docs. Here is Populator method:
and here is GetValueSets method with extra logging. Here I am adding all the content node's properties as index field:
Log messages from this method come through and I see them in Umbraco logs, but index is still empty and configured for 8 fields instead of many more. In Azure Search service - the same....... So still fighting..
In case anyone is interested in the topic, here is very basic Umbraco solution with custom ExamineX index configuration - github
Thank you, this helped! I'm evaluating ExamineX and my 100 allowed items were taken up by media items so I was able to exclude them from the ExternalIndex pretty easily.
Hi all - We will update the documentation for custom indexes today.
Its actually easier than what is listed above and you don't need to inherit from
UmbracoContentAzureSearchIndex
.For custom indexes with Examine, you can continue to inherit from the normal/documented index types based on the Umbraco docs https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/reference/searching/examine/indexing
Then, to opt your custom index into ExamineX (probably based on your environment type like 'prod'), you can just do:
For Umbraco Content based indexes (i.e. UmbracoContentIndex/IUmbracoContentIndex):
builder.Services .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>("YourIndexName")
For non-Umbraco custom indexes based on custom data sources (i.e. UmbracoExamineIndex/IUmbracoIndex):
builder.Services .AddExamineXAzureSearchIndex<AzureSearchIndex>("YourIndexName")
Docs updated for custom indexes https://examinex.online/customization#custom-indexes
I am really struggling to get this to work. I followed the umbraco code to the letter and have it working perfectly with local indexes.
I have a custom index which only needs a few custom properties based on the content being indexed (such as converting a location to lat/long for example).
However when I try to switch it to an ExamineX index (which are working for Internal/External indexes) I get a load of different issues:
Can I confirm whether the existing line of config should be removed....or should both lines exist? The documentation could be interpreted either way.
Should any other code changes be required to swap from a local to azure index using ExamineX? A full code example would be very useful, that works both when building an index and publishing content with custom fields being added in the valuesetbuilder.
The ExamineX extension methods should not be done within a composer.
Ordering is important and composers execute randomly within the call to
services.AddUmbraco(...)...AddComposers()
The ExamineX call to AddExamineXAzureSearch() on the Umbraco builder since ExamineX version 6 was done for a reason - because composers are difficult to manage if there are cross dependencies between them and ordering is important.
As an example, your startup code should look something like
Within AddExamineXAzureSearch(), it does this:
So if you are using the AddExamineXAzureSearchIndexFromUmbraco for a custom lucene based index, call this after the call to AddExamineXAzureSearch
I can look into adding a full code example to the ExamineX docs.
Hi Shannon, I will try moving these round however a full code sample would be much appreciated!
Just to create a basic custom index (with nodeid, alias and including a few derived fields, does not need the full set of umbraco properties) and to configure this with examineX so it works reliably when doing an index rebuild and publishing content.
Thanks
You should remove the AddExamineLuceneIndex call and only leave the ExamineX call. There are 2 different ExamineX calls depending on whether your custom index is of type UmbracoContentIndex/IUmbracoContentIndex (AddExamineXAzureSearchIndexFromUmbraco) or UmbracoExamineIndex/IUmbracoIndex (AddExamineXAzureSearchIndex).
Thanks, that's what I'd originally tried but just get the errors/issues mentioned.
I'm assuming it's the 'umbraco' one I need but it seems to just use the full set of umbraco content properties when publishing a node.
Using the 'custom' one breaks the back office Examine dashboard.
Hmm OK sorry I don't have any experience with UmbracoContentIndex, only the UmbracoExamineIndex and then adding my values from scratch.
I am however going down the same path as you (creating a custom index based on a content index but with extra values), but I haven't gotten to it yet :)
@Shannon I am helping Sue work through this issue, not sure if its just the application in general but i stripped the index creation back as we have spoken before about the setup. As we have also discussed, if its not working on Azure then stripping it back to the local indexes is a good place to start, which i have done but having a very odd issue.
We use the setup below in a couple of different Umbraco installs and its fine, so its very odd, but we are not getting any feedback from Examine / Umbraco to understand if anything is failing as if it is, its doing it silently.
Basically we have:
Startup has this code:
The WebsiteIndexOptions Gets fired, the WebsiteIndexerComponent gets fired and i can see it registering the event handler but whilst the index is created, it has no records and no fields.
If we try and publish the index, nothing happens (we had the same issue with ExamineX Azure for this setup, no logging apart from the initial index rebuild notification).
But, if i publish a record with a document type that is allowed to be indexed by this custom index, it actually hits the TransformingIndexValues method and adds that record to the index.
Its like the index is being provisioned but not correctly wired up to anything, but the system knows about it because publishing an item pushes that item into the index. We have had issues before with nothing firing, but i don't think we have ever had an issue where it only fires for publishing. I may attempt to create a blank Umbraco project and point to the database and just migrate the index code, but is there any specific reason why this could be the case?
Yeah I'd suggest maybe starting with a new site and keep things as simple as possible. Depending on what tier you are using, you also might be hitting Azure Search limitations. If these limits are breached in any way (even in the payload), the indexing fails silently unfortunately. https://github.com/SDKits/ExamineX/issues/5
More than happy to assist everyone to ensure you get this working. FYI I'm away next week so if I don't reply that is why.
Might also be worth asking on the ExamineX GitHub support since we don't normally monitor this forum that closely.
is working on a reply...