Examine indexes are different between Dev and Production
I have made some changes to my searching page. In dev, everything is working as planned. I'm having issues when I push the changes to production.
Dev environment is Win10Pro, VS2017, IIS Express.
Production environment is WinServer2012R2, IIS8
When I run searches in Prod, I get a "Key is not present in dictionary" exception.
Both dev and prod are pointed at the same database (SQL Server 12)
I copied the entire root directory from dev to prod, so all the config files are exactly the same.
Code:
var searcher = ExamineManager.Instance.SearchProviderCollection["ExternalSearcher"];
var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.Or);
var query = searchCriteria.Field("nodeName", s)
.Or()
.Field("postContent", s)
.Or()
.Field("postTags", s)
.Or()
.Field("postAuthor", s)
.Compile();
var searchResults = searcher.Search(query);
foreach (var searchResult in searchResults)
{
var sb = new StringBuilder();
foreach (var searchResultField in searchResult.Fields)
{
sb.AppendLine($"{searchResultField.Key}");
}
return Content(sb.ToString());
}
This foreach just goes into the first result and returns all the fields that are return in the search result. This is just a test to see what's different.
Again, the examine config files are exactly the same.
When I look at the /AppData/TEMP/ExamineIndexes/External/Index directory, there are massive differences.
IN DEV, there are 55 files.
53 begin with _3 and are TVX, FDT, FDX, FNM, FRQ, etc files
2 are the segments files.
IN PROD, there are 3 files.
0.cfs
segments.gen
segments2
I"m not sure what other info you would need.
Is there ANY reason why the environments would create vastly different indexes? The code files are exactly the same. An exact copy of the entire directory from DEV to PROD, including bin. I've tried deleting the External index directory and it is always created the same way with the same 3 files.
I think the main reason of this issue is that you have 2 instances of Umbraco pointing to the same database. So in fact you have a load balanced environment.
So when you publish something on dev the new conten will get indexed. The prod environment however doesn't know this change is done and the index there will not be updated.
Going in to the developer section and then try to rebuild the examine index in the Examine dashboard will probably solve your issue because it will reindex all the content.
Thank you for the reply. Unfortunately, this did not change any of the reported behavior. I had forced the reindexing earlier by just deleting the contents of the index folder and watching it rebuild.
BUT! Your reply did point me to the developer section that I had forgot about having all the Examine Management features. That did give me some more insight.
Your suggestion led me to the Examine Management section which pointed me to more things to dig into. I found my solution as explained below. I'm posting it here, so I can mark it as the solution.
I think I may have answered my own question here.
I'm not sure what triggers it, but Umbraco/Examine/Lucene must detect that it is running on IIS as opposed to IIS Express, or something like that, to determine if it needs to optimize the index.
When it optimizes, it seemingly removes all the fields that aren't explicitly included in the ExamineIndex.config file.
I manually updated that file to ensure that all the fields I needed were explicitly added to the IndexAttributeFields and IndexUserFields section of the ExternalIndexSet and now I'm all good.
I'm learning as I go along here! Thanks again for your help!
After Dave pointed me to the Examine Management area in the backoffice (that I had completely forgot existed) I did notice some info that might be useful.
IN DEV:
Documents in index = 772
Fields in index = 63
Has deletions = true (8)
Optimized = false
IN PROD:
Documents in index = 769
Fields in index = 15
Has deletions = false (0)
Optimized = true
System fields and User fields match exactly.
Provider Properties match exactly EXCEPT CommitCount. In dev it is 0 and in production it is 775 and the LuceneIndexFolder is different, but correct for each environment.
All these values are under the ExternalIndexer section under Indexers.
Still baffled! What could make the Fields in index so drastically different?
Examine indexes are different between Dev and Production
I have made some changes to my searching page. In dev, everything is working as planned. I'm having issues when I push the changes to production.
Dev environment is Win10Pro, VS2017, IIS Express. Production environment is WinServer2012R2, IIS8
When I run searches in Prod, I get a "Key is not present in dictionary" exception.
Both dev and prod are pointed at the same database (SQL Server 12)
I copied the entire root directory from dev to prod, so all the config files are exactly the same.
Code:
This foreach just goes into the first result and returns all the fields that are return in the search result. This is just a test to see what's different.
IN DEV:
id key parentID level writerID creatorID nodeType template sortOrder createDate updateDate nodeName urlName writerName creatorName nodeTypeAlias path allowComments featuredImageUrl postAuthor postContent postPublishedDate postTags wordPressId _NodeId _IndexType _Path _NodeTypeAlias _Key _Icon _RawpostContent
IN PROD:
id nodeName nodeTypeAlias parentID postTags postContent _NodeId _IndexType _Path _NodeTypeAlias _Key _Icon _RawpostContent
Again, the examine config files are exactly the same.
When I look at the /AppData/TEMP/ExamineIndexes/External/Index directory, there are massive differences.
IN DEV, there are 55 files. 53 begin with _3 and are TVX, FDT, FDX, FNM, FRQ, etc files 2 are the segments files.
IN PROD, there are 3 files. 0.cfs segments.gen segments2
I"m not sure what other info you would need.
Is there ANY reason why the environments would create vastly different indexes? The code files are exactly the same. An exact copy of the entire directory from DEV to PROD, including bin. I've tried deleting the External index directory and it is always created the same way with the same 3 files.
Thanks if advance!
Trent
Hi Trent,
I think the main reason of this issue is that you have 2 instances of Umbraco pointing to the same database. So in fact you have a load balanced environment.
So when you publish something on dev the new conten will get indexed. The prod environment however doesn't know this change is done and the index there will not be updated.
Going in to the developer section and then try to rebuild the examine index in the Examine dashboard will probably solve your issue because it will reindex all the content.
Dave
Dave,
Thank you for the reply. Unfortunately, this did not change any of the reported behavior. I had forced the reindexing earlier by just deleting the contents of the index folder and watching it rebuild.
BUT! Your reply did point me to the developer section that I had forgot about having all the Examine Management features. That did give me some more insight.
Thank you!
Dave,
Your suggestion led me to the Examine Management section which pointed me to more things to dig into. I found my solution as explained below. I'm posting it here, so I can mark it as the solution.
I think I may have answered my own question here.
I'm not sure what triggers it, but Umbraco/Examine/Lucene must detect that it is running on IIS as opposed to IIS Express, or something like that, to determine if it needs to optimize the index.
When it optimizes, it seemingly removes all the fields that aren't explicitly included in the ExamineIndex.config file.
I manually updated that file to ensure that all the fields I needed were explicitly added to the IndexAttributeFields and IndexUserFields section of the ExternalIndexSet and now I'm all good.
I'm learning as I go along here! Thanks again for your help!
Trent
After Dave pointed me to the Examine Management area in the backoffice (that I had completely forgot existed) I did notice some info that might be useful.
IN DEV:
Documents in index = 772
Fields in index = 63
Has deletions = true (8)
Optimized = false
IN PROD:
Documents in index = 769
Fields in index = 15
Has deletions = false (0)
Optimized = true
System fields and User fields match exactly.
Provider Properties match exactly EXCEPT CommitCount. In dev it is 0 and in production it is 775 and the LuceneIndexFolder is different, but correct for each environment.
All these values are under the ExternalIndexer section under Indexers.
Still baffled! What could make the Fields in index so drastically different?
Thanks again!
Trent
is working on a reply...