Sign in Register
Go to solution
Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Chris Kim 48 posts 254 karma points c-trib
    Apr 05, 2018 @ 22:04
    Chris Kim
    0

    Examine index empty / incomplete

    -- Umbraco v7.5.3, Examine 0.1.83 (also tried 0.1.89)

    We have a fairly large website (about 70,000 nodes) that we have added two new examine indexes to.

    All is fine in our dev environment, which only has about 400 nodes, but once deployed to our UAT environment (a fairly recent copy of production), one of the two indexes is incomplete (5658 indexed out of 28839) and the other is empty (should contain 559 documents).

    I am able to reproduce this locally with a copy of the database. Running in debug, the log shows the following for the incomplete index rebuild (triggered from Developer section in Umbraco):

     2018-04-06 09:14:35,601 [P8728/D2/T127] DEBUG Umbraco.Web.UmbracoModule - Begin request: http://localhost:62543/umbraco/BackOffice/Api/ExamineManagementApi/PostRebuildIndex?indexerName=ArticleIndexer.
 2018-04-06 09:14:35,632 [P8728/D2/T127] DEBUG Umbraco.Core.Sync.DatabaseServerMessenger - Syncing from database...
 2018-04-06 09:14:35,671 [P8728/D2/T127] DEBUG Umbraco.Core.Sync.DatabaseServerMessenger - Complete (took 41ms)
 2018-04-06 09:14:35,686 [P8728/D2/T127] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Rebuilding index, Provider=ArticleIndexer, NodeId=-1
 2018-04-06 09:17:37,551 [P8728/D2/T6] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index deleted for term: __IndexType with value content, Provider=ArticleIndexer, NodeId=-1
 2018-04-06 09:17:37,557 [P8728/D2/T6] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index created for node 4483, Provider=ArticleIndexer, NodeId=4483
 2018-04-06 09:17:37,616 [P8728/D2/T6] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index created for node 6013, Provider=ArticleIndexer, NodeId=6013
[... 5654 more lines for other documents ...]

 2018-04-06 09:19:05,087 [P8728/D2/T6] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index created for node 79599, Provider=ArticleIndexer, NodeId=79599
 2018-04-06 09:19:05,103 [P8728/D2/T6] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index created for node 79611, Provider=ArticleIndexer, NodeId=79611
 2018-04-06 09:19:05,103 [P8728/D2/T6] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index deleted for term: __IndexType with value media, Provider=ArticleIndexer, NodeId=-1
 2018-04-06 09:19:06,683 [P8728/D2/T120] DEBUG Umbraco.Web.UmbracoModule - Begin request: http://localhost:62543/umbraco/BackOffice/Api/ExamineManagementApi/PostCheckRebuildIndex?indexerName=ArticleIndexer.
 2018-04-06 09:19:06,683 [P8728/D2/T120] DEBUG Umbraco.Core.Sync.DatabaseServerMessenger - Syncing from database...
 2018-04-06 09:19:06,717 [P8728/D2/T120] DEBUG Umbraco.Core.Sync.DatabaseServerMessenger - Complete (took 24ms)
 2018-04-06 09:19:06,720 [P8728/D2/T120] WARN  Umbraco.Web.WebServices.ExamineManagementApiController - Property value was null when setting up property on indexer: ArticleIndexer property: DirectoryFactory

    I have looked at the node with ID 79611 and the next one after in the umbraco.config, but didn't spot anything extraordinary.

    For the empty index, the log shows the following:

     2018-04-05 16:48:28,576 [P8728/D2/T114] DEBUG Umbraco.Web.UmbracoModule - Begin request: http://localhost:62543/umbraco/BackOffice/Api/ExamineManagementApi/PostRebuildIndex?indexerName=AuthorIndexer.
 2018-04-05 16:48:28,676 [P8728/D2/T114] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Rebuilding index, Provider=AuthorIndexer, NodeId=-1
[... some unrelated lines I believe, scheduled publishing, GetRemainingTimeoutSeconds etc]
 2018-04-05 16:52:13,642 [P8728/D2/T95] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index deleted for term: __IndexType with value content, Provider=AuthorIndexer, NodeId=-1
 2018-04-05 16:52:13,642 [P8728/D2/T95] DEBUG UmbracoExamine.DataServices.UmbracoLogService - Index deleted for term: __IndexType with value media, Provider=AuthorIndexer, NodeId=-1
 2018-04-05 16:52:49,175 [P8728/D2/T128] DEBUG Umbraco.Web.UmbracoModule - Begin request: http://localhost:62543/umbraco/BackOffice/Api/ExamineManagementApi/PostCheckRebuildIndex?indexerName=AuthorIndexer.

    These are the two index sets we have added:

      <IndexSet SetName="ArticleIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Article/">
    <IndexAttributeFields>
      <add Name="id" />
      <add Name="nodeName"/>
      <add Name="nodeTypeAlias" />
    </IndexAttributeFields>
    <IndexUserFields>
      <add Name="intro"/>
      <add Name="pageTitle"/>
      <add Name="openGraphTitle"/>
      <add Name="Author"/>
    </IndexUserFields>
    <IncludeNodeTypes>
      <add Name="Article"/>
    </IncludeNodeTypes>
  </IndexSet>

  <IndexSet SetName="AuthorIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Author/">
    <IndexAttributeFields>
      <add Name="id" />
      <add Name="nodeName"/>
      <add Name="nodeTypeAlias" />
    </IndexAttributeFields>
    <IndexUserFields>
      <add Name="hideOnWebsite"/>
    </IndexUserFields>
    <IncludeNodeTypes>
      <add Name="Author"/>
    </IncludeNodeTypes>
  </IndexSet>

    And the Indexers and Searchers:

      <add name="AuthorIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"/>
  <add name="ArticleIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"/>

  <add name="ArticleSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" />
  <add name="AuthorSearcher" type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine" />

    Note: the internal index contains all nodes - it does take a long time to build the index (20-40 minutes depending on environment), but at least it works.

    Any pointers would be highly appreciated.

    Copy Link
  • Nigel Wilson 945 posts 2077 karma points
    Apr 06, 2018 @ 01:38
    Nigel Wilson
    0

    Just an idea - have you tried deleting the nodes (delete and then empty recycle bin) - so delete 79611, rebuild index, then delete the next node, rebuild.

    The above might confirm if there was an issue with a specific node.

    Cheers, Nigel

    Copy Link
  • Chris Kim 48 posts 254 karma points c-trib
    Apr 06, 2018 @ 03:11
    Chris Kim
    0

    Hi Nigel,

    Thanks for the response and the idea. Unfortunately that didn't get me any further, after deleting 79611 and the next node, it just stopped two nodes later.

    And my other index still ended up empty.

    Copy Link
  • Shannon Deminick 1526 posts 5272 karma points MVP 3x
    Apr 06, 2018 @ 04:35
    Shannon Deminick
    100

    I'd recommend not having too many indexes, the work required to build each one will be 100% the amount of work re-done as a previous index. Each index building does all of the data lookups independently so basically for each index you add, you increase the time and resources it will take to re-index items and rebuild indexes. You can quite easily use the External Index which contains all published values and just specify the correct search criteria.

    I know this doesn't solve your issue but just wanted to mention this.

    I guess this might be of interest, but not sure

    Property value was null when setting up property on indexer: ArticleIndexer property: DirectoryFactory

    Always best to use the latest Examine version though. Umbraco 7.5.3 is quite old and I'm sure there's been updates and fixes since then regarding the data lookups to populate the indexes. You should try to upgrade locally to vLatest and see what happens then.

    Copy Link
  • Chris Kim 48 posts 254 karma points c-trib
    Apr 09, 2018 @ 06:28
    Chris Kim
    0

    Hi Shannon,

    Thanks for your response!

    I upgraded to 7.5.14 and this fixed the issue, plus made the indexing much faster! (I noticed this ticket regarding indexing speed: http://issues.umbraco.org/issue/U4-9420)

    With the indexes generating faster, this should actually solve some of our other problems we're having with this site.

    For now, I only did an upgrade within the minor version to avoid breaking changes, but we're also having discussions about a bigger upgrade with the client.

    Thanks also for your comment regarding multiple indexes. Because the indexes were always so slow to build, we weren't using the external index at all. Knowing that multiple indexes are not recommended (and with the significant speed improvements after the upgrade), we will probably rework the indexes above and just use the external index instead.

    Cheers

    Copy Link
Please Sign in or register to post replies

Write your reply to:

Draft