Hello,

We have noticed that our site has become very slow to start due to indexing, in particular the size of the indexes. Inspecting the indexes, I have noticed that some fields hold duplicate data, and these fields are the largest in the record set. See below for example screenshot (I have edited the html to reduce the markup size for screenshot)

This is happening in both Internal and external indexes, resulting in a Examine folder size that is well over a Gigabyte in size.

Now I can remove some fields in the external index by overriding the ContentValueSetValidator as follows:

 //remove any '__Raw*' and '*.Row' values to reduce duplicate data being indexed
foreach (var publishField in valueSet.Values.Where(x => x.Key.StartsWith("__Raw") || x.Key.EndsWith(".Row")).ToList())
{
    valueSet.Values.Remove(publishField.Key);
    isFiltered = true;
}

and this works well, but I cannot find how to achieve this for the internal index. I have scoured the Umbraco.Examine source code, and cannot see a way to do this.

Is there a way to do this, or is there a better way to achieve this?

Thanks