That does indeed look like the same goals. I would love to take a stab at making an Elasticsearch provider, if I can find the branch containing these changes.

I don't like that approach, because that moves the responsibility for the behaviour from Core to the provider, which means that behaviour would change when changing provider.

Also, each provider would basically duplicate the logic for when/what to index, which does not seem like it should be required for implementing a provider.

I guess a middle ground could be making a BaseIndexer in core that you could inherit to just make a simple indexer for a different data store. Then that base would contain all the default behaviour.

In any case, it all depends on what Shannon is currently doing for V8, and what direction they want to go with it.

I think the provider model should be as simple as possible, so that any provider should be able to support all back office features without additional configuration. This ensures site builders can swap provider, without being an expert in the underlying engine for a given provider.

If the site implementation then needs geo search, or other more complex features, then the implementation should just go straight to that search engine and not through the provider. I don't see a need for the provider interface to provide anything more than simple text search capabilities for the frontend.

Each provider could also just have it's own set of config options to enhance the indexing of data, such as language specific analyzers etc. But I don't think that part should be abstracted into the core config.

Core config should probably just be the list of indexes and fields/doctypes to index. That would cover 90% of all usecases I think. Then providers can provide additional config options for cases that need more tweaking.

Hi Shannon

V8 sounds a lot better, especially that we get rid of Examine for media queries, and the whole indexing on Startup.

I havn't had a play with the ISearchableTree yet.

in V7: the issue with the raw lucene queries inside umbraco is more the fact that they include specific fields in the queries, witch means that the implementer either has to replace the fields in the raw query or have the exact same mapping as Examine for fields in their index, and that kind of forces the way the implementation has to look, this is where I would like the abstractions to go, to avoid this kind of binding.

ISearchableTree is a very simple interface, it just takes some search text. It might be extended for list view searches at some stage but it will remain a simple interface, no field requirements, etc...

Umbraco v8 only constructs raw lucene queries for it's own ISearchableTree implementations to pass to Examine - but since you can replace ISearchableTree that should be ok, and since the 3 main hosted Examine providers that I want to see working (Azure Search, Elastic Search and Solr) will support this syntax and will automatically be indexing the correct fields, it should work for those without having to replace ISearchableTree. Potentially these queries can be built with ISearchCriteria instead but I'm not sure yet.

It should still be possible to replace Examine entirely if you really want to so long as you don't use the UmbracoHelper.Search method tied to ISearchCritieria and you don't need to know about fields being searched on etc...

Confirming this is possible is part of the current work being done.

It would be great if the core ISearhableTree implementations did no raw lucene queries. Then the examine provider would work as a single abstraction for search.

I will definitely give it a go with an Elasticsearch provider.

Wouldn't a raw lucene query make assumptions about the structure of the document in Elasticsearch?

Another question: How will the new language versions be handled with regard to indexing? Will there be an index per language? Are the fields going to be prefixed? Will a query contain the intended language?

Yes, but Examine is in charge of indexing and searching so the correct stuff would go into the index to match any of the searches. Even if ISearchCriteria is used assumptions are made that there are certain fields and data that need to exist. If you want a totally different/custom index structure, etc.. then you'll need to replace ISearchableTree and keep your own index based on your own events.

Variants/languages are interesting. For the first iteration, all languages will be indexed into a single document with suffixed names (i.e. nodeName_en-us, nodeName_es-es) Each field can have it's own analyzer.

Down the road we might decide to have an index per language or have it as an option, but it would be possible to configure separate indexes per language on your own if you wanted.

There are pros/cons to each of these but we've chosen a single index because this is easier to implement for now and it more easily allows contextual searching. What a single index doesn't allow for is to show a search result row per language whereas multiple indexes would allow that and vice-versa.

Will a query contain the intended language

If you are referring to ISearchableTree then at the moment the plan is to not have that, however based on my work next week that could change so stay tuned for updates on that. As far as Examine goes, searching for values via languages is just done by searching on specific language fields.

I was mostly thinking of the search helpers used by site builders for the frontend. So if a user is browsing the german site, then any calls to IPublishedContentQuery.Search would then get the de-de postfixes added to the field names before the query is sent to the examine provider?

Hmm, my reply seems to have gone missing :D

I was mostly thinking of when users are searching on the site. So if a visitor is on the german site, will calls to IPublishedContentQuery.Search automatically get the de-de postfixes added before hitting the Examine provider?

Ah right, yup something like that would make sense, i haven't got around to that API yet but i'll make a note about that, thanks!

Hi Shannon

I was just wondering what the status is on this? I forked the current temp8 repo/branch, and as far as I can see, the LuceneIndex is still used directly from Umbraco?

Is it still the plan to remove that hard dependency, to enable other Examine providers to be used?

I've now also spent some time with the Examine V1.0 branch.

The index provider interface now seems cleaner. No more xml, which is nice :)

When looking at the IQuery interface, for creating supporting the fluent query creation, it started getting harder to follow the logic. There is a whole lot of logic going on with regards to nesting and/or/not queries.

Taking on the task of implementing an Elasticsearch version of IQuery seems like a rather large chunk of work. I would be reluctant to start that task before also having created a test suite that proves that a query would deliver the same results across providers.

There also seems to be a broken abstraction where the IBooleanOperation inherits IOrdering, which means that you can add sorting on a nested boolean query, which does not really make sense?

I completely understand the idea of abstracting the query from the engine, but I do wonder if it adds enough value, as opposed to the complexity it adds? Would it make sense to make a simpler shared query interface, that should work across providers, and let and more advanced queries be done with provider specific queries?

All IQuery usages in Core could be isolated in a replaceable search facade in Umbraco Core, which could then be implemented by engine specific versions.

Packages would need to restrict themselves to the simpler search interface, but I don't know how many packages are utilizing advanced search currently?

These are just some thoughts from diving a bit further into the task. I don't think I would be able to complete an Examine provider as fast as I would have liked to :)

Some answers inline:

Taking on the task of implementing an Elasticsearch version of IQuery seems like a rather large chunk of work. I would be reluctant to start that task before also having created a test suite that proves that a query would deliver the same results across providers.

Sure it's a bit of a process to implement, search is complicated ;) A test suite would be near impossible to make because any implementation internally will create a different query mechanism. The lucene index will create a lucene query, an elasticsearch provider might not, it might build up some other query to be sent with the rest request, or maybe it will just stick with a lucene query, not sure.

There also seems to be a broken abstraction where the IBooleanOperation inherits IOrdering, which means that you can add sorting on a nested boolean query, which does not really make sense?

Good catch! I will update this. This is in-flux at the moment, the interfaces were being mashed together to come up with a nicer fluent syntax (i.e. no longer having a strange Compile method or being able to order by in the middle of a query)

I completely understand the idea of abstracting the query from the engine, but I do wonder if it adds enough value, as opposed to the complexity it adds? Would it make sense to make a simpler shared query interface, that should work across providers, and let and more advanced queries be done with provider specific queries?

Not really sure what you are expecting here tbh. You want to be able to implement a simple search functionality with no bells and whistles but why? Maybe you are thinking along the lines of having generic implementation like: ISearcher + ISearcher<TQuery> where TQuery:IQuery and IQuery is a stripped down version of what it currently is, then you'd have LuceneSearcher<LuceneQuery> . I have made a POC for this a couple of years ago and it ended up being extremely complex with generics, but lots has changed since then so might still be viable ... is this what you are referring to?

There are 2 things: Examine and Umbraco, by default Umbraco will use Examine and Examine itself is an abstraction. In Umbraco the 'simple' abstraction that can be replaced is ISearchableTree. The only place Examine APIs are referenced publicly is in the 2 overload methods: IPublishedContentQuery.Search with IQueryExecutor. It will be possible to replace the IPublishedContentQuery implementation too if you wanted which means you can wire up anything behind the simple search overload method that accepts a string term. If you want to replace the simple back office search you can easily with ISearchableTree. If you then want to use your own search functionality on your front-end you can do whatever you want but then you are also in charge of indexing which is currently an ugly task since you'll have to replicate all of the event logic.

I have a WIP branch in Examine to implement an Azure Search provider which is partially working. This re-uses much of the lucene searching logic because it also just builds up a lucene query, see the branch: https://github.com/shazwazza/examine/tree/azure-search-1.0

Ultimately I would want the implementation to be easier, part of the problem is the current implementation is more complex than it needs to be but I don't have time to un-wind all of that and much of it is many years old. The azure search branch starts simplifying it a little bit (as far as the azure search provider goes)

I'm not sure I have any good suggestions that are realistic to do before v8 is released.

I think my main issue is that I find a fluent interface, such as IQuery, really hard to implement. If I were to reinvent search in Umbraco, I think I would prefer that the core expose a ISearchProvider interface, with methods such as:

IndexContent(IContent content)
IndexPublishedContent(IPublishedContent publishedContent)    
IndexMediaContent(IMedia media)

And for searching something like

SearchContent(string searchTerms,  string[] doctypes, string[] properties)    
SearchPublishedContent(string searchTerms,  string[] doctypes, string[] properties)
SearchMediaContent(string searchTerms,  string[] mediatypes, string[] properties)

I think that kind of interface would be enough to support the indexing and searches performed by core, as well as cover 80% of sites implementations when it comes to searching site content.

It would also be quite easy to implement providers for different search engines, regardless of if they were based on lucene or some cloud based API.

And if you need to do more complex queries on a site, just go directly to the provider you're using on the site and use whatever DSL they have exposed for querying their platform.

I realize this is not a small change, but just wanted to explain what I meant by having a simpler mandatory interface. I think the IQuery interface is too complex for most of us to write an implementation for. And I do think there is a need for non-file based search providers.

I think i mentioned above about the indexing part, i agree that it would be nice to expose an interface for indexing... and that can still be possible in 8.x but there's no time to do that at the moment. The interface wont be as simple as you suggest though ;) there is a lot to consider when indexing based on what happens within umbraco like whether an entire branch needs to be removed, updated, etc... but in any case an interface like that would be good.

For searching ... i feel like i'm repeating myself ... you can just replace the ISearchableTree's for any of the trees and it's a very simple interface.

That would cover all searching that is done within umbraco. Then on your front-end you can do whatever you want.

Implementing IQuery can be a great thing because it will just happily work with Umbraco directly without modifying anything, but like I've said before you are not forced to do this. With ISearchableTree and potentially the indexing interface stuff mentioned above that will get you to where you want to be

Even without the indexing interface mentioned here, you can still implement an Examine index provider to handle the indexing - that's a reasonably easy thing to implement. Then you have your indexing working with your own provider. For search if you don't implement an Examine search provider with IQuery, you can throw a NotImplementException if you wanted and then replace the ISearchableTree with your own, and on your front-end do whatever you want.

Yes ISearchableTree is just a simple interface and you can replace it for any tree.

I'm just trying to give you some options that you 'could' do. I'm not saying that you should just go ahead and make half of an Examine provider ... but i am saying that you could.

Each tree can have a different search method - just like we ship with, some trees search via the DB, some search via Examine. We can't just create a single interface to search all trees, and we have to consider that developers have their own trees. You don't want to replace all searching for all trees all of the time, that would be worse than "replacing X nubmer of ISearchableTree implementations"

Ideally we have an indexing interface baked into the CMS that you can implement to index your own data and then replace whatever ISearchableTree you need to (in this case, the only ones using Examine are content, media, members IIRC)

I also don't know what this means?

It would be nice for example to also have an interface for the Examine Dashboard features. Supporting reindexing etc. for any provider.

We do have that, reindexing button is decided based on an IIndexPopulator, other things that configure the dashboard come down to the provider itself and if it implements IIndexDiagnostics.