On further investigation, it seems the root cause is that the Xml returned from library.GetMedia().ToXml() as used by ExamineBackedMedia, doesn't contain the url to the media item as an attribute. ExamineBackedMedia assumes the url to the item will be available in a property called "umbracoFile", however this property is not available in the xml.

library.GetMedia(1040, true).Current.OuterXml

Ok, so to me it looks like the main issue is that the Xml generated by a Media node does not contain any custom properties, unlike the Xml generated by a Document node.

Document overrides the XmlGenerate() method and serialises any custom properties as xml child nodes, however Media just defers to XmlGenerate provided by the Content class, which does not serialise any custom properties. Maybe the solution is to move the serialisation of custom properties up into the Content class, instead of the Document class???

Another effect of this is that the custom media properties (such as umbracoFile) are not indexed by Examine, as the UmbracoMediaService examine provider ultimately uses the same Xml provided by Media.XmlGenerate() (which as described above does not contain any custom properties).

So given that neither of the Xml datasources used by DynamicMedia include any custom properties (including umbracoFile), I can't see how DynamicMedia can access important properties such as Url (or the underlying umbracoFile). This seems like a fairly major issue for media handling with Razor in 4.7.1

Hi Greg

That doesn't seem right.. since Library.GetMedia is what you'd typically use from XSLT anyway.
Could I get a SQL DB backup and site files so I can investigate with the debugger? If it's a webmatrix/sqlce site then I only need the zip of the whole site root (which includes the CE database)

You upgraded from 4.7.0 right, not from < 4.6.0 (which could indicate an XML conversion issue between schemas - the old schema is probably not supported by razor)

Might be worth double checking what XSLT schema is in use, that's in umbracoSettings or web.config since if it's the old schema, that's /data[@alias='umbracoFile'] yeah I can see that not working, I didn't even consider that when writing the feature. 

Perhaps a developer changed it to the old schema for compatibility reasons?

It's odd the media nodes aren't in examine though, they should be - though yes, custom properties aren't there anyway.

Media URLs definitely work in a stock install, so something odd must be going on with the data here

Happy to help but I'll need the backup, put it on dropbox or similar and DM me the link on twitter. I'm busy tonight 2011-09-27, evening, NZ (gmt+13) time, but I will investigate it tomorrow during the day

Hi Gareth,

Thanks for getting in touch. Yes its an upgrade from 4.7.0 using the new 4.5+ schema. I've just performed a test in a freshly installed site, and as you suggest things seem to be working as expected, so it looks like I jumped the gun to blame the new release - apologies!!!

<LATER> 

Ok, I've managed to track down where things have gone wrong in my site - somehow the content of the cmsContentXml table for those media nodes doesn't reflect the correct xml structure (ie: no property nodes are present in the xml). After resaving the media nodes in the backend, the cmsContentXml table is updated to reflect the correct structure - and subsequently the DynamicMedia has access to all of the properties and begins to work as expected. I'm not sure how the cmsContentXml got corrupt (I'm pretty sure its not a manual intervention thats caused it) - its a bit concerning that it can get into this state, but I've burnt enough time on this already, so fingers crossed it doesn't reoccur.

Anyway, I'm back on track now. Thanks again for your response, and I'm sorry for the false alarm.

Cheers,
Greg

In the case that anyone is still using the old schema, but still want razor to access the properties see the post here:http://allan-laustsen.blogspot.com/2012/04/umbraco-razor-dynamicmedia-umbracofile.html

Thanks for the update & patch, Allan.
Nice catch - I didn't consider legacy schema when implementing that class.

I am aiming to get a few fixes like this into 4.7.2 so I will assign the codeplex issue to myself.

Gareth

With legacy schema off.. and post upgrading to 4.7.2 umbracoFile returns nothing when trying:

var audioWelcome = Library.MediaById(audioMediaId);

audioWelcome.umbracoFile

OR

audioWelcome.UmbracoFile (neither return anything) but if you try @audioWelcome it returns umbraco.MacroEngines.DynamicMedia and i know it is populated but not sure how to get what xml is returned as the razor macro is inline..

What version did you upgrade from Tom?

If you upgraded from a version that was using legacy schema, there's a bit of a gotcha in that umbraco.library.getMedia (that is used for custom properties or when the examine index doesn't contain the media item [due to it being broken or not updated yet]) sometimes can end up with it's xml store (cmsContentCache table, I think) containing both legacy and non legacy schema XML

The fix for my sites that I have upgraded is clearing that table

Gareth

I upgraded from 4.7.1 to 4.7.2  turns out i needed to go in and save the media types in the settings section and then it automagically began working!

I am wondering how Greg was able to check the WasLoadedFromExamine property ... The DynamicMedia item i get returned from @Library.MediaById doesn't seem to expose the DynamicBackingItem publicly (which also neither exposes the ExamineBackedMedia publicly) but maybe I can answer my own question by assuming the only way to check this is to run from source code on localhost?

Thank you!

To #Tom & #Funka!

If you already have added media nodes to your umbraco, there is properly an issue with the examine indexes in your installation. You need to install the Examine Dashboard: http://our.umbraco.org/projects/developer-tools/examine-dashboard
And then rebuild the indexes using the dashboard, then you dont have to re-save all the media items.

@Funka - probably by attaching a debugger using the correct sourcecode version to a binary installation.
You can then step through the code.

I think the solution here may be to clear the cmsContentXml table and then rebuild the examine indexes.

There is no way to bypass the fallback to GetMedia - it does this because the custom properties are not in examine (which was identified in this thread as being a problem in the XML generation - I just thought it was by design)

I have no problem if you want to add in a config variable to control this behaviour but it won't help here since the data appears to not be available from the GetMedia call or Examine.

When the DynamicMedia class (actually DynamicNode now) gets a media item, if it WasLoadedFromExamine, when you access a property which is non standard (e.g. not id,path etc), it calls library.GetMedia anyway - due to the custom properties not being available.

Gareth

Hi Gareth,

Thank you for the reply. The problem I am experiencing in 4.7.2 is that the HTML tags are being stripped out from my custom (richtext editor) properties on Media items. Interestingly, Media items I created back in a 4.7.1 installation work just fine, but after upgrading to 4.7.2 any new Media items I create do NOT work.

Looking through the database (in both [cmsContentXml], [cmsPropertyData]) the HTML tags are all there intact, but when renderend, only the media items originally created under 4.7.1 are keeping the HTML tags.  Thus why I am wondering how to debug whether it is due to any changes in the Examine engine between these two versions.

Here's the thread I orignally raised this issue (it has no solutions yet) : Richtext editor property on a Media type - HTML tags are stripped when rendered?

And another related thread (also with no solution) : Media item html tags stripped off in Razor and escaped in xslt

Thank you!

Would it be possible to obtain a site exhibiting the problem for debug?

Gareth

Hello Gareth,

What is the best way to deliver this to you? (Is there a way to privately contact you through these forums?) We have several 4.7.2 sites this is turning out to be a problem on. It might be easier to just have you try to recreate this issue in a blank 4.7.2 installation.

(1) add a new property called "description" to the built-in "Image" Media Type. It should be a richtext editor.

(2) add a new Image and add some markup to it, for example, multiple paragraphs, some bold text and a hyperlink somewhere.

(3) Try and get the markup to display on a page using Razor, e.g., @Library.MediaById(1234).description

(4) Notice that all of the HTML tags (the bolding and the hyperlink) have been stripped.

If you are not able to reproduce this problem, let me know how to contact you and will send you some login credentials to our of our dev sites.

Thank you!

Those reproduction instructions will be fine.

I'll try and check this out and come back to you on this thread.

Gareth

I have just attempted to reproduce this issue on a fresh webmatrix 4.7.2 with SQL CE.

1) Created a document type and template
2) Added description RTE to the image type
3) Added a test image
4) Created a new razor script with macro
5) Created a test page using the document type and published it
6) Displayed the image description using @Library.MediaById

I got HTML outputting fine.

Here's the SQLCE test site: https://dl.dropbox.com/u/2923715/mediatest.admin.password.96MU548TqPao3b.7z

Admin password is in the filename

Gareth

Hello again Gareth.

I have downloaded the test project you provided, thank you!  Upon running it initially, it did work as expected. However, it soon "broke" and went back to the problem I've noted. I've re-extracted your sample project a few times and can consistently recreate the problem.

Try this:

Edit the sample media item. Add a second paragraph to the description. Save it.

Then reload the homepage a few times. (Interestingly for me, it took maybe 10 seconds after saving the media item before I could see the updated description text.)

Notice the HTML and link are all stripped out. Even trying to put the description back to one paragraph doesn't fix the problem. Please let me know I am not going insane and that you are able to recreate this problem??!?

Thank you!

I've recreated this again and again, and have found as soon as you add that second paragraph, it will "break" things.

My best guess, is that something thinks this description field should be XML, but there isn't one root node: there are instead two sibling <p> nodes. (Which of course is not well-formed so perhaps if something is trying to parse this as XML behind the scenes then we run into this problem?)

Your example image only had one paragraph in its description, so the single <p> in there was able to act as a single root node if my theory about XML parsing errors is correct, and thus why you didn't initially see the same problem I am guessing.

Thank you!

Hi Funka

I've inspected the code closely and have determined that when the ExamineBackedMedia class falls back to umbraco.library.GetMedia then everything is fine, there is HTML delivered from the property.
The same applies when there is only a few nodes in the original RTE, in this case it will be one of two cases
1) Examine has indexed the data as a plain string, without stripping the HTML
2) Examine hasn't yet indexed the media data and the code falls back to GetMedia (which yeilds HTML) 

There seems to be a scenario where the media item is loaded direct from examine and the description field is contained within the examine SearchResult class in the IDictionary<string,string> Fields property.

When this dictionary contains the markup for the description, all the HTML has been stripped - it's only plain text.

Also- another potential curveball, when a property is loaded from examine, we don't get the DataTypeId - the editor that is used for the data.

DynamicNode uses this to apply the formatting and, for example, ensure that a HtmlString is returned (or a int, DateTime, etc) - that's not actually a bug though, I've just made it use Guid.Empty here - presumably because at the time of testing, the custom properties *weren't* in the examine SearchResult and it was never an issue.

TLDR: Looks like it's either an examine bug or by design. I will see if I can get someone else to review this and discuss with me.

Gareth

Thank you Gareth, I look forward to hearing what you find and thank you for looking into this.

If I were to throw my two cents in, I would vote for it being a bug, since "by design" is still flawed since sometimes it works (retains html) and sometimes it doesn't (strips html), with no real clear difference as to why.

There is one other thing in your post that I fond interesting and had a wrong assumption about, which is that only base/intrinsic properties on Media items were cached by Examine, and all custom properties were looked up from the DB each time. Thought I had read something to that effect somewhere, but in any case, and this bug notwithstanding, having the whole thing indexed/cached certainly seems preferable. (If only it worked as expected!)

Thanks again!

Hi Funka

Examine is doing this strip intentionally, but it wasn't a problem until recently because when I wrote the DynamicNode feature, the custom properties for media weren't available in the index.

So, it's by design in examine, but a bug in DynamicNode.

This thread, http://our.umbraco.org/forum/developers/api-questions/17737-Examine-returns-wrong-markup#comment66705 that I was pointed to on twitter explains the behaviour and how to work around it when you're using your own custom indexer.

Unfortunately, that doesn't help us and this should be raised as a bug in DynamicNode.

I don't know the exact fix here though,  it seems that I need to alter the native indexes to have secondary copies of the RTE content that isn't stripped of it's formatting.
I am just using the built in examine indexes that ship with Umbraco and I haven't altered them before.

I will drop a line out on the core mailing list and see if someone knows what we need to do. It is too late for 4.8 beta but we may  be able to get a fix into the 4.8 RTM

Gareth

The 4.8 RTM is already out so this will probably have to wait for 4.9.

Jeroen

Hi again Gareth and Jeroen,

Just wanted to let you know I've created an issue on CodePlex for this { http://umbraco.codeplex.com/workitem/30983 } so that I could stay in the loop on when this might be addressed, am very hopeful this will be able to make it into 4.9.

Thanks again for your attention to this!

EDIT:

Just for posterity, issue was migrated to YouTrack site as issue U4-644.

This comes to mind again because today I am having new but similar problem. This time, it is reading from a uComponents UrlPicker data type---which stores its information as XML. (And also which is stripped, just like my HTML, so thus impossible to use.)

Thank you!

Hi Funka!

Have a chat to Sebastian, he might be able to find your issue.

I have posted to the google group about it a few weeks ago but I can't fix this one by myself, need someone more familiar with examine to help.

Gareth

is working on a reply...