how to fix xml data integrity issues with media content - Using Umbraco and getting started

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 14:52

How to fix XML Data Integrity issues with Media & Content?

Does anyone know how to fix these XML Data Integrity issues with Media and Content? I've already dumped AppData --> cache/courier/Logs/preview/Temp folders, and I've already rebuilt the ExternalIndexer and InternalIndexer in Examine Management.

I think there used to be a "Fix" button for these but I'm not sure / can't see it on Umbraco 7.8.1.

enter image description here

Any ideas?

Best, Chris

Edit: Not sure if it's related but I'm getting hundreds of these errors in UmbracoTraceLog:

     2018-03-12 09:04:56,460 [P15328/D3/T26] INFO  Umbraco.Web.WebServices.ExamineManagementApiController - Rebuilding index 'InternalMemberIndexer'
 2018-03-12 09:04:56,464 [P15328/D3/T26] INFO  UmbracoExamine.DataServices.UmbracoLogService - Rebuilding index, Provider=InternalMemberIndexer, NodeId=-1
 2018-03-12 09:04:56,529 [P15328/D3/T26] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - Start data queries - member, Provider=InternalMemberIndexer, NodeId=-1
 2018-03-12 09:04:56,634 [P15328/D3/T26] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - End data queries - member, took 104ms, Provider=InternalMemberIndexer, NodeId=-1
 2018-03-12 09:04:56,646 [P15328/D3/T34] INFO  Umbraco.Web.WebServices.ExamineManagementApiController - Rebuilding index 'InternalMemberIndexer' done, 1 items committed (can differ from the number of items in the index)
 2018-03-12 09:05:00,743 [P15328/D3/T28] WARN  Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache - Failed (10 times) to retrieve medias from Examine index and had to load them from DB. This may indicate that the Examine index is corrupted.
 2018-03-12 09:05:03,827 [P15328/D3/T34] INFO  Umbraco.Web.WebServices.ExamineManagementApiController - Rebuilding index 'InternalIndexer'
 2018-03-12 09:05:03,827 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Rebuilding index, Provider=InternalIndexer, NodeId=-1
 2018-03-12 09:05:03,869 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - Start data queries - content, Provider=InternalIndexer, NodeId=-1
 2018-03-12 09:05:04,142 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - End data queries - content, took 272ms, Provider=InternalIndexer, NodeId=-1
 2018-03-12 09:05:04,142 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - Start data queries - media, Provider=InternalIndexer, NodeId=-1
 2018-03-12 09:05:04,179 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - End data queries - media, took 35ms, Provider=InternalIndexer, NodeId=-1
 2018-03-12 09:05:04,235 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1063
 2018-03-12 09:05:04,322 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1065
 2018-03-12 09:05:04,325 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1068
 2018-03-12 09:05:04,340 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1082
 2018-03-12 09:05:04,341 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1084
 2018-03-12 09:05:04,342 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1085
 2018-03-12 09:05:04,343 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1093
 2018-03-12 09:05:04,344 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1094
 2018-03-12 09:05:04,345 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1095
 2018-03-12 09:05:04,516 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1096
 2018-03-12 09:05:04,520 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1118
 2018-03-12 09:05:04,523 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1139
 2018-03-12 09:05:04,525 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1163
 2018-03-12 09:05:04,526 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1252
 2018-03-12 09:05:04,527 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1256
 2018-03-12 09:05:04,528 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1260
 2018-03-12 09:05:04,528 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1264
 2018-03-12 09:05:04,529 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1268
 2018-03-12 09:05:04,530 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=1273

I'm also getting these errors for the ExternalIndexSet:

 2018-03-12 09:05:06,002 [P15328/D3/T34] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "InternalIndexSet". Please ensure all names are unique, Provider=InternalIndexer, NodeId=3102
 2018-03-12 09:05:06,007 [P15328/D3/T34] INFO  Umbraco.Web.WebServices.ExamineManagementApiController - Rebuilding index 'InternalIndexer' done, 1370 items committed (can differ from the number of items in the index)
 2018-03-12 09:05:06,760 [P15328/D3/T40] INFO  umbraco.BusinessLogic.Log - Log scrubbed.  Removed all items older than 2018-03-11 09:05:06
 2018-03-12 09:05:12,803 [P15328/D3/T28] INFO  Umbraco.Web.WebServices.ExamineManagementApiController - Rebuilding index 'ExternalIndexer'
 2018-03-12 09:05:12,804 [P15328/D3/T28] INFO  UmbracoExamine.DataServices.UmbracoLogService - Rebuilding index, Provider=ExternalIndexer, NodeId=-1
 2018-03-12 09:05:12,854 [P15328/D3/T28] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - Start data queries - content, Provider=ExternalIndexer, NodeId=-1
 2018-03-12 09:05:12,923 [P15328/D3/T28] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - End data queries - content, took 68ms, Provider=ExternalIndexer, NodeId=-1
 2018-03-12 09:05:12,923 [P15328/D3/T28] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - Start data queries - media, Provider=ExternalIndexer, NodeId=-1
 2018-03-12 09:05:12,925 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1063
 2018-03-12 09:05:12,958 [P15328/D3/T28] INFO  UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - End data queries - media, took 35ms, Provider=ExternalIndexer, NodeId=-1
 2018-03-12 09:05:12,967 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1601
 2018-03-12 09:05:12,968 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1726
 2018-03-12 09:05:12,969 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1778
 2018-03-12 09:05:12,971 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1727
 2018-03-12 09:05:12,973 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1896
 2018-03-12 09:05:12,974 [P15328/D3/T6] INFO  UmbracoExamine.DataServices.UmbracoLogService - Field "template" is listed multiple times in the index set "ExternalIndexSet". Please ensure all names are unique, Provider=ExternalIndexer, NodeId=1992

Copy Link

Alex Skrypnyk 6150 posts 24110 karma points MVP 8x admin c-trib

Mar 12, 2018 @ 15:19

Hi Chris

You are right, there should be "Fix" button, and this button should fix this issue

Do you have some issues with files and folders permissions? Or database?

Alex

Copy Link

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 15:28

Here's the contents of ~\Umbraco\Views\dashboard\developer\healthcheck.cshtml; doesn't seem to have any "Fix button" code:

<div ng-controller="Umbraco.Dashboard.HealthCheckController as vm">

    <div ng-if="vm.viewState === 'list'">

        <div class="flex justify-between items-center">
            <h3 class="bold">Health Check</h3>
            <button class="umb-era-button -green" type="button" ng-click="vm.checkAllGroups(vm.groups);">Check All Groups</button>
        </div>

        <div class="umb-healthcheck-help-text">
            <p>The health checker evaluates various areas of your site for best practice settings, configuration, potential problems, etc. You can easily fix problems by pressing a button.
            You can add your own health checks, have a look at <a href="https://our.umbraco.org/documentation/Extending/Healthcheck/" target="_blank" class="btn-link -underline">the documentation for more information</a> about custom health checks.</p>
        </div>

        <div class="umb-healthcheck">

            <div class="umb-air" ng-repeat="group in vm.groups" ng-click="vm.openGroup(group);">
                <div class="umb-healthcheck-group">

                    <div class="umb-healthcheck-title">{{group.name}}</div>

                    <div class="umb-healthcheck-group__load-container" ng-if="group.loading">
                        <umb-load-indicator></umb-load-indicator>
                    </div>

                    <div class="umb-healthcheck-messages" ng-hide="group.loading || !group.totalSuccess && !group.totalWarning && !group.totalError && !group.totalInfo">

                        <div class="umb-healthcheck-message" ng-if="group.totalSuccess > 0">
                            <i class="icon-check color-green"></i>
                            {{ group.totalSuccess }}
                        </div>

                        <div class="umb-healthcheck-message" ng-if="group.totalWarning > 0">
                            <i class="icon-alert color-yellow"></i>
                            {{ group.totalWarning }}
                        </div>

                        <div class="umb-healthcheck-message" ng-if="group.totalError > 0">
                            <i class="icon-delete color-red"></i>
                            {{ group.totalError }}
                        </div>

                        <div class="umb-healthcheck-message" ng-if="group.totalInfo > 0">
                            <i class="umb-healthcheck-status-icon icon-info"></i>
                            {{ group.totalInfo }}
                        </div>

                    </div>

                </div>
            </div>

        </div>

    </div>

    <div ng-if="vm.viewState === 'details'">

        <umb-editor-sub-header>
            <umb-editor-sub-header-content-left>
                <a class="umb-healthcheck-back-link" href="" ng-click="vm.setViewState('list');">&larr; Back to overview</a>
            </umb-editor-sub-header-content-left>
        </umb-editor-sub-header>


        <div class="umb-healthcheck-group__details">

            <div class="umb-healthcheck-group__details-group-title">
                <div class="umb-healthcheck-group__details-group-name">{{ vm.selectedGroup.name }}</div>
                <button class="umb-era-button -white -text-black" type="button" ng-click="vm.checkAllInGroup(vm.selectedGroup, vm.selectedGroup.checks);">Check group</button>
            </div>

            <div class="umb-healthcheck-group__details-checks">

                <div class="umb-healthcheck-group__details-check" ng-repeat="check in vm.selectedGroup.checks">

                    <div class="umb-healthcheck-group__details-check-title">
                        <div class="umb-healthcheck-group__details-check-name">{{ check.name }}</div>
                        <div class="umb-healthcheck-group__details-check-description">{{ check.description }}</div>
                    </div>

                    <div class="umb-healthcheck-group__details-status" ng-repeat="status in check.status">

                        <div class="umb-healthcheck-group__details-status-icon-container">
                            <i class="umb-healthcheck-status-icon icon-check color-green" ng-if="status.resultType === 0"></i>
                            <i class="umb-healthcheck-status-icon icon-alert icon-alert color-yellow" ng-if="status.resultType === 1"></i>
                            <i class="umb-healthcheck-status-icon icon-delete icon-delete color-red" ng-if="status.resultType === 2"></i>
                            <i class="umb-healthcheck-status-icon icon-info icon-info" ng-if="status.resultType === 3"></i>
                        </div>

                        <div class="umb-healthcheck-group__details-status-content">

                            <div class="umb-healthcheck-group__details-status-text">
                                <div ng-bind-html="status.message"></div>
                                <div ng-if="status.description" ng-bind-html="status.description"></div>
                            </div>

                            <div class="umb-healthcheck-group__details-status-actions" ng-if="status.actions">
                                <div class="umb-healthcheck-group__details-status-action" ng-repeat="action in status.actions">
                                    <ng-form name="healthCheckAction">

                                        <div ng-if="action.valueRequired">
                                            <div><label class="bold">Set new value:</label></div>
                                            <input name="providedValue" type="text" ng-model="action.providedValue" required val-email/>
                                        </div>

                                        <button
                                            type="button"
                                            class="umb-era-button umb-button--s -green"
                                            ng-class="{ '-inactive': healthCheckAction.providedValue.$invalid}"
                                            ng-click="vm.executeAction(check, $parent.$index, action);"
                                            ng-disabled="healthCheckAction.providedValue.$invalid">
                                            {{action.name}}
                                        </button>

                                    </ng-form>

                                    <div class="umb-healthcheck-group__details-status-action-description" ng-if="action.description" ng-bind-html="action.description"></div>
                                </div>
                            </div>

                        </div>

                    </div>

                    <div ng-show="check.loading">
                        <div class="umb-healthcheck-group__details-status-overlay"></div>
                        <umb-load-indicator></umb-load-indicator>
                    </div>

                </div>

            </div>

        </div>

    </div>

</div>

Copy Link

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 15:41

I also dumped the entire contents of ~\Umbraco\ folder and replaced from a 7.8.1 download and can't see the fix button.

Is there a SQL query or something I can run to manually fix this? Or how can I proceed to fix?

Copy Link

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 15:20

Also tried emptying the Media trash and deleting App_Data\umbraco.config and let the site rebuild it. Still have XML Data Integrity errors.

Copy Link

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 15:23

I'm primarily having issues with Content ID's and Media ID's mismatching between staging and production. Because of this, I've been having to delete and re-upload things on production and have temporarily abandoned staging + courier to transfer updates. A mapped ID might not match up to anything or it might match up to an old content or media item.

Copy Link

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 15:48

Update: -- Partially Fixed

I tried hitting {website.com}/umbraco/dialogs/republish.aspx?xml=true and got a YSOD. Then I tried again and it worked.

This results in the content ID's being fixed; but the Media Items are still messed up. Is there an equivalent republish for Media?

enter image description here

Copy Link

Richard Hamilton 79 posts 169 karma points

Nov 26, 2018 @ 16:18

Click on the green square?

Copy Link

Chris Van Oort 110 posts 371 karma points

Mar 12, 2018 @ 17:33

Opened a bug for the Fix button not being visible: http://issues.umbraco.org/issue/U4-11088

Update / Fix Partial PEBCAK here, but if I click the Green square that's lacking any text it does actually POST to execute the action to fix the Media and fix the Content.

Couple other details in case someone else stumbles upon this:

How the data integrity checks work: https://our.umbraco.org/forum/umbraco-7/using-umbraco-7//73079-xml-cache-data-integrity-check-in-730-what-is-it-and-why-does-the-fix-throw-an-exception#comment-238271

Where the backend code is in case you want to dig into it: https://github.com/umbraco/Umbraco-CMS/blob/dev-v7/src/Umbraco.Web/HealthCheck/Checks/DataIntegrity/XmlDataIntegrityHealthCheck.cs

Client app/Backoffice code: https://github.com/umbraco/Umbraco-CMS/tree/dev-v7/src/Umbraco.Web.UI.Client/src/views/dashboard/developer

It just ends up posting to /umbraco/BackOffice/Api/HealthCheck/ExecuteAction with all of the required cookies, HTTP headers, etc. and a JSON payload like this:

{"name":null,"alias":"checkMediaXmlTable","healthCheckId":"d999eb2b-64c2-400f-b50c-334d41f8589a","actionParameters":null,"description":null,"valueRequired":false,"providedValue":null}

What that actually does in SQL is still a bit of magic to me though.

Copy Link

Laura Weatherhead 25 posts 153 karma points MVP 6x c-trib

Nov 27, 2018 @ 12:31

Hey Chris,

I know this is a super old ticket, but the missing "fix" text is still missing in the later versions on Umbraco (e.g. 7.12.4) and now that it's been pointed out I can't unsee it!!

Put in a wee PR to fix it here: https://github.com/umbraco/Umbraco-CMS/pull/3782 think it was just a missing name property. Linked it to your issue as well :)

Cheers,

Laura

Copy Link

Chris Van Oort 110 posts 371 karma points

Nov 27, 2018 @ 15:07

Appreciate it -- you rock!

Copy Link

is working on a reply...

Flag this post as spam?

How to fix XML Data Integrity issues with Media & Content?