Open discussion about the Umbraco 'repository pattern' to define re-usable content/data
I’m hoping my ramble will be of interest to many as it’s an aspect of Umbraco that's been bugging me for some time.
I have diligently followed the common “repository pattern” whereby I define a document type for fragments of data that I wish to record once. These fragments are created at root level inside a "repo" folder so that they can be picked on various pages typically using an MNTP.
Examples of the types of data you might want to create once, but use many times in a document might be:
Details about Businesses.
Details about People.
Promotional display items i.e panels, accordions and other re-usable "widgets" or UI.
Forms.
Definitions, Tags or Types which allow you to define filtering/grouping in lists of items.
There are two reasons I particularly dislike this method...
The repository content exists in the document node tree and has a route attached to it even though it will never have a template or be called in a browser. Accidental calls to the url have to be redirected or 404’d. Personally, I don’t like this as it seems counterintuitive to the requirement and use of these data fragments.
When you use an MNTP picker to pick a repo item, the data doesn’t exist in that document, only the id to it. This requires extra code to look the content up and make it available for the document/template.
Has anyone found a better solution to this pattern?
I’ve looked a little at Fluidity and UI-O-Matic, both on the surface look promising. For these be a true solution though I would need fine-grained user access rights to the custom data not just the entire section and the data should exist in the document when its linked, no MNTP style look-ups.
In my minds eye, I was thinking of an extension to nested content and document type compositions whereby you tag the composition as a data store off the document tree. You would define these elements in a dedicated section (like in Fluidity/UI-O-matic) with a FULL user access UI (section and fine-grained permissions i.e. CRUD on folder containers and individual items).
If you pick this special composition inside a nested content datatype, it will act more like an MNTP and give you a picker instead but with the difference that you can (optionally) edit the fields with a warning it would be a universal change (where used) or detached and become normal nested content (just this page). On publish the data fragment is embedded (retaining any links) as part of the page content like a normal nested content.
Yes I have had similar thoughts as you with "data" being stored in the content tree.
However in terms of admin UI for non technical users, and coupled with the caching of site content (app_data/umbraco.config) I guess the simplest solution is the one we currently have.
I guess one possibility would be split out "data" content into a separate cache, but then that introduces other issues, and possibly performance issues too.
No easy solution to resolve I guess, but good idea to post your thoughts to get some discussion going.
The reasons you list disliking this approach don't seem to be all that inconvenient. I would ask, is it really worthwhile to invest much time in address it?
Ok here's another reason that could easily cause irritation for content editors who wouldn't differentiate the difference between genuine content routes and "embedded data"
Content without a route cannot be previewed! Clicking on the preview button pops up the preview window and it hits you with your lovely 404 page. If you've added an internal redirect it cycles forever and displays nothing!
Content without a route cannot be previewed! Clicking on the preview button pops up the preview window and it hits you with your lovely 404 page.
A simple solution to that particular problem would be to disable the preview button and "Links" area in the back office for any node that does not have a template. Perhaps it could be a setting (e.g., in case there are people who use altTemplate or something of that sort for pages that don't have a default template set for a given node).
One simple way to get around this is to create a redirect template for this kind of content. For instance, if this content actually belongs on the parent page then just create a template that has:
Response.Redirect(Model.Content.Parent.Url);
This also has advantage that if they click the URL in Properties tab it works, too.
Yes i've used that in the past for items that have parent containers, but it's not appropriate for repository data as this has no natural parent that would be a page.
There's an open issue for this - but there are some wrinkles with making this the default as it could be a breaking change for people who use alternative templates etc.
Was thinking about this last night - would a solution be to have a third option when creating a doctype, where we currently have new document type and new document type without a template.
A new type with no template or route which exists only for building out repositories of content (or nested content items, etc), so has no routing, no template, no previewability and so on, purely a container for data.
It makes sense the a picker would only store the id, otherwise you'll end up with content referenced in a node that has to be updated when the source node is modified - fetching the content by id avoids that.
Yes a third option makes a lot of sense, but i wonder if relation linking might be way around a picker containing no more than the id? With relation linking you could then update all nodes using the one item.
Depending on the size and the use case, and it won't work with the preview issue... Look at it as a performance optimization. Not very content editor friendly.
What we sometimes do is adding additional fields to the document in the Examine cache. These additional fields can contain data from other nodes. This makes it unnecaissary to retrieve the additional nodes.
I do like the idea of having data fragments as an umbraco concept. As the default content finders don't need to take these into account.
There is/was a plan to implement something called Schemas, these are essentially doc types without any routing features, so effectively content fragments.
There was an attempt to get this in before Nested Content was merged in, as these Schemas would work perfectly for that too, but in the end we went for just getting it in there as was. I'm not sure the roadmap on Schemas but I still think they are very much needed for things like Nested Content but also to allow features like this where we have data holding content but shouldn't be directlty routable to.
Many interesting thoughts. Some concerns can be solved easily today. Others need to be discussed. But managing out-of-tree data is definitively a topic. Traveling to DK today so will post a longer reply sometime later!
As I said, many interesting thoughts. Going to try to address some here.
Some very general thoughts:
Many sites end up managing "out of tree" data so the problem is a fairly common one.
The more we can use what already exists (like, document types, manage these fragments in the tree, permissions, etc) the easiest it is to maintain.
The general solution is to have "out of tree" data somewhere in the tree (repository node pattern) and then pick with MNTP or another picker. It could be that a dedicated, more appropriate picker, could exist for that type of usage. I am not sure. But... it can be done, it's "just" a new property editor.
Getting the picked content
Then, there is the "the picker knows the ID but requires extra code to get the actual data" problem. By default, the picker will tell you that you have picked node 1234 and you have to go get it. But, that is already fixed. Or at least, we already have tools to fix it. If you use the proper proprety converter for the picker, you can have the lookup performed in the property converter. The end result would be that GetPropertyValue("picked") would not return an integer (1234) but directly an IPublishedContent instance containing the picked item. Or, an enumeration of picked items.
This works today. I might need to explain it in more details in another post, but all websites I've done and use pickers, combined with ModelsBuilder, end up doing eg @Model.Picked.Title and it just works.
Editing
About "local edit in the picker"... two things:
Editing the picked content would require a custom picker property editor. Not a small task, but can be done.
Overriding the picked content... is an entirely different beast. I have no idea how it could be done--would require more thinking.
Routing
The problem with routing is that these nodes should not be routed at all. Basically, they should not be treated as "tree items" at all: no routing, no preview, no nothing.
There is no ideal solution, today. And I do have looked for some, for sites I have done.
I feel it would be totally fine to look into how we could flag some document types as "fragments" and then Core would know they should not be routed, previewed, nothing. I feel it could be relatively easy (i.e. cheap) to implement, at least in v8. I'd love to see it done.
Rendering
When the content cache returns out-of-tree items... well, because the content cache maintains the tree, they are going to be IPublishedContent instances (ie have a parent, children, etc). It may be slightly confusing.
Version 8 introduces IPublishedElement. We haven't talked much about it, but basically, an IPublishedElement is a content type + values for properties, and that is all. And IPublishedContent implements IPublishedElement and adds a bunch of tree-related stuff (template, parent, etc).
We could make it so that the picker casts the returned item to IPublishedElement so that it's less confusing. For instance, Intellisense would not detect a Parent property.
But of course, ideally, the property converter should make sure that the property value is a strongly typed model of some sort, eg IEmployeeBio that can be used directly.
Now what
Stopping here else it's going to be a very long post. Happy to answer questions and keep brainstorming.
If we want this discussion to turn into things that happen... we have to get organized. I have created this issue on the tracker, where I would love to list action items, ie things that can be "done".
Please discuss either here or on the tracker. Well, maybe it's better to discuss here, and just list actual action items on the tracker, OK? And then... let's see where this goes.
A content picker has the ability to be single or multi but also any number of doctypes too. This means you always get back an IEnumerable of IPublishedContent
At this point forward I've either had to cast the first item to the type I "expect" or use an @helper(type) method and pass a dynamic object to process all of the possible document types to be rendered.
It's still much better than GetPropertyValue("alias")
You will get back an enumerable of IPublishedContent indeed. Since items can be of different doctypes.
The "easy" case is when they however implement a common set of properties and that is what you are interested in: these properties can belong to a composition, and then you could get an IMyProperties instance.
When you really want to treat each doctype differently, well... what you get is an IPublishedContent but it is actually implemented by actual types such as MyClass1 and MyClass2 -- in which case I have found the @helper((dynamic) item) pattern to be both simple and elegant.
But the problem here is i'm trying to cast an IPublishedContent and this cannot be done. So the easiest way around this (in a single line) is to spice it up with a little Linq Lambda.
MemberFirm memberFirm = Model.MemberFirm.Select(x => new MemberFirm(x)).FirstOrDefault();
Which then gives me access to its properties.
Then onto the IEnumerable...
Initially I thought I would have to cast it too, at first I tried this...
IEnumerable<MemberContact> memberContact = memberFirm.Contacts.Select(x => new MemberContact(x));
So far so good...but then I thought, hang on, surely the types are also available for iteration as you've mentioned.
So I tried this...
foreach (MemberContact contact in MemberFirm.Contacts) { .... }
And it just worked! Whoop! No extra cast required.
ModelsBuilder really helps in simplifying and visualising the backoffice schema in code and to a certain extend it does ease the difficulty in implementing a repository pattern.
Regarding flagging some document types as "out-of-tree"... If you want to brainstorm this... We probably want to broaden the scope. With NuCache in version 8 it will be way easier to, for example, exclude some nodes from cache. Or other subtleties.
It may be that we want a way to manage a set of "flags" on document types, that could be used to tell Umbraco... many things about the document type. And I am wondering whether they could also be used at user-level.
What I have in mind is... I did some site where we would have some logic to see if a document type had this special iAmSpecialproperty type and that would tell use something, well, special, about every document of that type. So that instead of testing on document type aliases, we could check for the property.
It looks like it would be way more elegant to be able to do... content.ContentType.HasFlag(Flags.IsSpecial) and make decisions based upon this.
That's absolutely AWESOME! Exactly the kind of response I was after. I really do feel that it's a common pattern and a common headache in Umbraco.
I will take another look at your comments in "Getting the picked content", I'm using ModelsBuilder in two projects and I don't recall the property converter returning the actual content with a simple MNTP. I have used Nested content to return complete models through the document type comp. though.
Hi Martin, You make a very valid point about non-page data and page data - In fact the CMS that i designed and have used for 8 years splits content into two in this very way, with typical data stored in the non-page section as "People", "Investments", "Sectors", "News". In my case when i wanted to show these non-page items as a path - e.g. a person page, or an investment page, there was some automatic routing that added the item path onto the page that was "hosting" it and so forth.
We could make it so that the picker casts the returned item to
IPublishedElement so that it's less confusing. For instance,
Intellisense would not detect a Parent property.
I think traversing the tree should still be possible, since there are actually parent/child elements and it would feel strange that you cannot traverse the tree but there is a tree structure.
Is the question more, Should repeatable content/data be in the content tree at all?
In UI-o-Matic and Fluidity you define your data in a new, separate panel. If this idea was to be part of the core, you would use document types as you would with content but maybe mark them "special" in some way so that they can only be used in the data panel.
I don't see any reason why this data couldn't still be complex and recursive. Built out using a tree structure in the same way as content in the content tree. Or simple and flat depending on need.
I do feel it should be separated from the content tree in some way though.
Open discussion about the Umbraco 'repository pattern' to define re-usable content/data
I’m hoping my ramble will be of interest to many as it’s an aspect of Umbraco that's been bugging me for some time.
I have diligently followed the common “repository pattern” whereby I define a document type for fragments of data that I wish to record once. These fragments are created at root level inside a "repo" folder so that they can be picked on various pages typically using an MNTP.
Examples of the types of data you might want to create once, but use many times in a document might be:
Details about Businesses. Details about People. Promotional display items i.e panels, accordions and other re-usable "widgets" or UI. Forms. Definitions, Tags or Types which allow you to define filtering/grouping in lists of items.
There are two reasons I particularly dislike this method...
The repository content exists in the document node tree and has a route attached to it even though it will never have a template or be called in a browser. Accidental calls to the url have to be redirected or 404’d. Personally, I don’t like this as it seems counterintuitive to the requirement and use of these data fragments.
When you use an MNTP picker to pick a repo item, the data doesn’t exist in that document, only the id to it. This requires extra code to look the content up and make it available for the document/template.
Has anyone found a better solution to this pattern?
I’ve looked a little at Fluidity and UI-O-Matic, both on the surface look promising. For these be a true solution though I would need fine-grained user access rights to the custom data not just the entire section and the data should exist in the document when its linked, no MNTP style look-ups.
In my minds eye, I was thinking of an extension to nested content and document type compositions whereby you tag the composition as a data store off the document tree. You would define these elements in a dedicated section (like in Fluidity/UI-O-matic) with a FULL user access UI (section and fine-grained permissions i.e. CRUD on folder containers and individual items).
If you pick this special composition inside a nested content datatype, it will act more like an MNTP and give you a picker instead but with the difference that you can (optionally) edit the fields with a warning it would be a universal change (where used) or detached and become normal nested content (just this page). On publish the data fragment is embedded (retaining any links) as part of the page content like a normal nested content.
All comments welcome…
Hi Martin
Yes I have had similar thoughts as you with "data" being stored in the content tree.
However in terms of admin UI for non technical users, and coupled with the caching of site content (app_data/umbraco.config) I guess the simplest solution is the one we currently have.
I guess one possibility would be split out "data" content into a separate cache, but then that introduces other issues, and possibly performance issues too.
No easy solution to resolve I guess, but good idea to post your thoughts to get some discussion going.
Nigel
Thanks Nigel
I was wondering if @thechiefunicorn head ever given this any thought?
M.
The reasons you list disliking this approach don't seem to be all that inconvenient. I would ask, is it really worthwhile to invest much time in address it?
Ok here's another reason that could easily cause irritation for content editors who wouldn't differentiate the difference between genuine content routes and "embedded data"
Content without a route cannot be previewed! Clicking on the preview button pops up the preview window and it hits you with your lovely 404 page. If you've added an internal redirect it cycles forever and displays nothing!
A simple solution to that particular problem would be to disable the preview button and "Links" area in the back office for any node that does not have a template. Perhaps it could be a setting (e.g., in case there are people who use altTemplate or something of that sort for pages that don't have a default template set for a given node).
One simple way to get around this is to create a redirect template for this kind of content. For instance, if this content actually belongs on the parent page then just create a template that has:
This also has advantage that if they click the URL in Properties tab it works, too.
Hi Dan
Yes i've used that in the past for items that have parent containers, but it's not appropriate for repository data as this has no natural parent that would be a page.
Should be part of the back office by default really as a good UX. No template = no http generated route = no preview button.
There's an open issue for this - but there are some wrinkles with making this the default as it could be a breaking change for people who use alternative templates etc.
See http://issues.umbraco.org/issue/U4-10740
Was thinking about this last night - would a solution be to have a third option when creating a doctype, where we currently have new document type and new document type without a template.
A new type with no template or route which exists only for building out repositories of content (or nested content items, etc), so has no routing, no template, no previewability and so on, purely a container for data.
It makes sense the a picker would only store the id, otherwise you'll end up with content referenced in a node that has to be updated when the source node is modified - fetching the content by id avoids that.
Hi Nathan
Yes a third option makes a lot of sense, but i wonder if relation linking might be way around a picker containing no more than the id? With relation linking you could then update all nodes using the one item.
You can actually disable preview for certain doctypes.
Have a look at this issue : http://issues.umbraco.org/issue/U4-8530
There is code example on how to disable preview for all nodes. But you can easily add a check on doctype there.
Dave
Hi Dave
Thanks for the pointer...I will use that, but it should be OOTB.
For me it shouldn't be out of the box.
The option to do this is there if you need it.
Dave
It is, maybe a little subjective? Arguments for and against I suppose.
But it's a code only solution and questioning architecture and general usability.
Hi Dave
I've just tried out the snippet for disabling a preview.
used a contentModel.ContentTypeAlias check and it worked great.
Thanks for the pointer. M.
Depending on the size and the use case, and it won't work with the preview issue... Look at it as a performance optimization. Not very content editor friendly.
What we sometimes do is adding additional fields to the document in the Examine cache. These additional fields can contain data from other nodes. This makes it unnecaissary to retrieve the additional nodes.
I do like the idea of having data fragments as an umbraco concept. As the default content finders don't need to take these into account.
There is/was a plan to implement something called Schemas, these are essentially doc types without any routing features, so effectively content fragments.
There was an attempt to get this in before Nested Content was merged in, as these Schemas would work perfectly for that too, but in the end we went for just getting it in there as was. I'm not sure the roadmap on Schemas but I still think they are very much needed for things like Nested Content but also to allow features like this where we have data holding content but shouldn't be directlty routable to.
Hi Matt,
I think this is called PublishedElement now in V8
https://github.com/umbraco/Umbraco-CMS/blob/temp8/src/Umbraco.Core/Models/PublishedContent/IPublishedElement.cs
Stéphane showed this during his talk at the UK Fest
Dave
Oh well that's a surprise! So it has been decided to build this in then, awesome!
I think this will be used in Nested Content in V8. But could be used for other scenario's as well
Dave
Many interesting thoughts. Some concerns can be solved easily today. Others need to be discussed. But managing out-of-tree data is definitively a topic. Traveling to DK today so will post a longer reply sometime later!
Thanks Stephan
Look forward to hearing from you.
M.
As I said, many interesting thoughts. Going to try to address some here.
Some very general thoughts:
The general solution is to have "out of tree" data somewhere in the tree (repository node pattern) and then pick with MNTP or another picker. It could be that a dedicated, more appropriate picker, could exist for that type of usage. I am not sure. But... it can be done, it's "just" a new property editor.
Getting the picked content
Then, there is the "the picker knows the ID but requires extra code to get the actual data" problem. By default, the picker will tell you that you have picked node 1234 and you have to go get it. But, that is already fixed. Or at least, we already have tools to fix it. If you use the proper proprety converter for the picker, you can have the lookup performed in the property converter. The end result would be that
GetPropertyValue("picked")would not return an integer (1234) but directly anIPublishedContentinstance containing the picked item. Or, an enumeration of picked items.This works today. I might need to explain it in more details in another post, but all websites I've done and use pickers, combined with ModelsBuilder, end up doing eg
@Model.Picked.Titleand it just works.Editing
About "local edit in the picker"... two things:
Routing
The problem with routing is that these nodes should not be routed at all. Basically, they should not be treated as "tree items" at all: no routing, no preview, no nothing.
There is no ideal solution, today. And I do have looked for some, for sites I have done.
I feel it would be totally fine to look into how we could flag some document types as "fragments" and then Core would know they should not be routed, previewed, nothing. I feel it could be relatively easy (i.e. cheap) to implement, at least in v8. I'd love to see it done.
Rendering
When the content cache returns out-of-tree items... well, because the content cache maintains the tree, they are going to be
IPublishedContentinstances (ie have a parent, children, etc). It may be slightly confusing.Version 8 introduces
IPublishedElement. We haven't talked much about it, but basically, anIPublishedElementis a content type + values for properties, and that is all. AndIPublishedContentimplementsIPublishedElementand adds a bunch of tree-related stuff (template, parent, etc).We could make it so that the picker casts the returned item to
IPublishedElementso that it's less confusing. For instance, Intellisense would not detect aParentproperty.But of course, ideally, the property converter should make sure that the property value is a strongly typed model of some sort, eg
IEmployeeBiothat can be used directly.Now what
Stopping here else it's going to be a very long post. Happy to answer questions and keep brainstorming.
If we want this discussion to turn into things that happen... we have to get organized. I have created this issue on the tracker, where I would love to list action items, ie things that can be "done".
Please discuss either here or on the tracker. Well, maybe it's better to discuss here, and just list actual action items on the tracker, OK? And then... let's see where this goes.
Hi Stephen
Sorry for this being a little off-topic....
A content picker has the ability to be single or multi but also any number of doctypes too. This means you always get back an IEnumerable of IPublishedContent
At this point forward I've either had to cast the first item to the type I "expect" or use an @helper(type) method and pass a dynamic object to process all of the possible document types to be rendered.
It's still much better than GetPropertyValue("alias")
If there's a simpler way....i'm all ears!
You will get back an enumerable of
IPublishedContentindeed. Since items can be of different doctypes.The "easy" case is when they however implement a common set of properties and that is what you are interested in: these properties can belong to a composition, and then you could get an
IMyPropertiesinstance.When you really want to treat each doctype differently, well... what you get is an
IPublishedContentbut it is actually implemented by actual types such asMyClass1andMyClass2-- in which case I have found the@helper((dynamic) item)pattern to be both simple and elegant.Better than a giant if/then/else.
Hi Stephen
Thanks for your feedback, most helpful. I decided to dig into this a little further on one of my simpler projects that requires a repo.
I have two data lists
On any given page I have to link a single MemberFirm and within a MemberFirm I can link any number of MemberContacts. Both pickers are MNTP's.
So onto the code for each picker, at first I thought I could do this...
But the problem here is i'm trying to cast an IPublishedContent and this cannot be done. So the easiest way around this (in a single line) is to spice it up with a little Linq Lambda.
Which then gives me access to its properties.
Then onto the IEnumerable...
Initially I thought I would have to cast it too, at first I tried this...
So far so good...but then I thought, hang on, surely the types are also available for iteration as you've mentioned.
So I tried this...
And it just worked! Whoop! No extra cast required.
ModelsBuilder really helps in simplifying and visualising the backoffice schema in code and to a certain extend it does ease the difficulty in implementing a repository pattern.
Thanks for your genius! Martin
Regarding flagging some document types as "out-of-tree"... If you want to brainstorm this... We probably want to broaden the scope. With NuCache in version 8 it will be way easier to, for example, exclude some nodes from cache. Or other subtleties.
It may be that we want a way to manage a set of "flags" on document types, that could be used to tell Umbraco... many things about the document type. And I am wondering whether they could also be used at user-level.
What I have in mind is... I did some site where we would have some logic to see if a document type had this special
iAmSpecialproperty type and that would tell use something, well, special, about every document of that type. So that instead of testing on document type aliases, we could check for the property.It looks like it would be way more elegant to be able to do...
content.ContentType.HasFlag(Flags.IsSpecial)and make decisions based upon this.Hi Stephen
That's absolutely AWESOME! Exactly the kind of response I was after. I really do feel that it's a common pattern and a common headache in Umbraco.
I will take another look at your comments in "Getting the picked content", I'm using ModelsBuilder in two projects and I don't recall the property converter returning the actual content with a simple MNTP. I have used Nested content to return complete models through the document type comp. though.
Hi Martin, You make a very valid point about non-page data and page data - In fact the CMS that i designed and have used for 8 years splits content into two in this very way, with typical data stored in the non-page section as "People", "Investments", "Sectors", "News". In my case when i wanted to show these non-page items as a path - e.g. a person page, or an investment page, there was some automatic routing that added the item path onto the page that was "hosting" it and so forth.
I think traversing the tree should still be possible, since there are actually parent/child elements and it would feel strange that you cannot traverse the tree but there is a tree structure.
Is the question more, Should repeatable content/data be in the content tree at all?
In UI-o-Matic and Fluidity you define your data in a new, separate panel. If this idea was to be part of the core, you would use document types as you would with content but maybe mark them "special" in some way so that they can only be used in the data panel.
I don't see any reason why this data couldn't still be complex and recursive. Built out using a tree structure in the same way as content in the content tree. Or simple and flat depending on need.
I do feel it should be separated from the content tree in some way though.
is working on a reply...