Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Dat Nguyen 7 posts 87 karma points
    Nov 07, 2023 @ 12:50
    Dat Nguyen
    0

    uSync.Complete / Publisher with Custom SQLFileSystem for mediafiles

    Hi,

    I am trying to setup uSync.Complete with our Custom Media FileSystem which stores media files in a SQL table as bytes for security reasons.

    A bit of context how it works.

    If you access the following URL:

    /umbraco/backoffice/api/mediafiles/tntpnmfh/picture1.jpg/
    

    Our Custom MediaController will query the mediafile from table and return it.

    My Challenge

    So far I am able to pull media nodes from one Umbraco instance using the Remote Media browsing. The media nodes are synched perfectly fine with right reference to the mediafile. However, the mediafiles itselves are not synched and stored into our table.

    My idea was to create a Custom Media Handler that could fetch the mediafile on the other instance on /umbraco/backoffice/api/mediafiles/... and store it in DB when using the Umbraco.Publisher.

    Does sound like the right approach? Or is there a better way?

    Br, Dat

  • Kevin Jump 2342 posts 14889 karma points MVP 8x c-trib
    Nov 07, 2023 @ 14:17
    Kevin Jump
    0

    Hi,

    If you have a custom filesystem (so IFileSystem / iFileStore) for your SQL media store, then uSync should 'just wortk'

    (core docs here : https://docs.umbraco.com/umbraco-cms/extending/filesystemproviders)

    uSync uses the media storage APIs so it then doesn't care about the underling filesystem - it doesn't pull the files of disk but via the fileprovider which does the IO

    I don't think replacing the media handler will actually work, because the media files are synced independently of the actual media items (so there is a point where the media handler reads the media stream to get a hash, so we can compare media items, but if they are different another bit of the publishing process goes off to the media services and gets the actual files for syncing.

    I will have a look, to confirm, but i think the IFileService way is probably the only real way if can be integrated into complete.

    Update: https://docs.umbraco.com/umbraco-cms/extending/filesystemproviders#creating-a-custom-file-system

  • Dat Nguyen 7 posts 87 karma points
    Nov 07, 2023 @ 14:50
    Dat Nguyen
    0

    Hi Kevin,

    Thanks for the quick reply.

    Our Custom SQL filesystem is implemented based on the docs you provided where it implements IFileSystem and it all works fine with Umbraco. ( https://docs.umbraco.com/umbraco-cms/extending/filesystemproviders#creating-a-custom-file-system)

    Based on your information then uSync shold be able to pull the mediafiles from fileprovider then from our SQL table if I understand correctly?

  • Kevin Jump 2342 posts 14889 karma points MVP 8x c-trib
    Nov 07, 2023 @ 15:59
    Kevin Jump
    0

    Hi Dat,

    yes, it should ... in theory

    during the sync process when media is required uSync will call MediaService.GetMediaFileContentStream which i believe in turn is calling the FileSystem from the mediaFileManager and doing FileSystem.FileExists and FileSystem.OpenFile calls

    so you should be able to see those calls getting hit in your custom provider.

    you can also turn the debugging up for uSync (see https://docs.jumoo.co.uk/usync/uSync/guides/debugging) you will then see some debug messages from the SyncMediaFileService around the process of getting a file (e.g "CopyMediaFile: {Id} {folder}" )

    and warning messages if uSync can't get to the media for whatever reason such as No media file stream to copy Media Id {Id}

    internally from our code. we calculate the path of the file from the value in the media item -

    • if debug is on that will be logged either as file path from crop: {path} or "file path from property: {path}

    • the value is then passed into the GetMediaFileContentStream call on MediaService (mediaService.GetMediaFileContentStream(localSource);

    • this will either return the stream or null (if the media service can't find a file at the right path). and then we log a warning if its missing.

    going the other way is very similar, we get the path and upload the file using SetMediaFileContent again if there are issues they should raise warnings in the logs.

  • Dat Nguyen 7 posts 87 karma points
    Nov 08, 2023 @ 06:46
    Dat Nguyen
    0

    Thanks Kevin.

    That should be enough to get me going :)

  • Dat Nguyen 7 posts 87 karma points
    Nov 08, 2023 @ 13:04
    Dat Nguyen
    0

    Hi again Kevin,

    I manage to boil down to the issue.

    After pulling an image from Umbraco Server Instance A all the syncfiles and mediafile are available at following path on Umbraco Intsance B

    C:\Users\xxx\source\repos\umbraco-cms-powerstation-prod\umbraco\Data\TEMP\uSync\receive\

    where my mediafile is located at:

    C:\Users\xxx\source\repos\umbraco-cms-powerstation-prod\umbraco\Data\TEMP\uSync\receive\

    Since I have a Custom SQL Media store my FileExists method checks if the file exist in my SQL media table rather than the physical file at

    ...\TEMP\uSync\receive....

    hence GetMediaFileContentStream will always return null and never upload the file via. the SetMediaFileContent method

    I need to somehow extend my fileExists method to also check the for synched files in ...\TEMP\uSync\receive....

    Do you know of any helper method where I can get some meta data such as path to the mediafile being synched?

  • Kevin Jump 2342 posts 14889 karma points MVP 8x c-trib
    Nov 08, 2023 @ 13:31
    Kevin Jump
    0

    Hi,

    I am not sure you have to know about the uSync folders.

    they are used temporarly when zipping and unzipping the media between servers, but uSyncs internal services then read these file into streams and that is what is passed to the media service for restore.

    so on the source server we write a file to the temp folder:

    • we first call the media service to get the stream

      mediaService.GetMediaFileContentStream(localSource)

    • then it is written to disk

      syncFileService.SaveFile(target, stream);

    and that is zipped up and sent overthe wire to the other server.

    at the other end : - we unzip the files to a temp folder - we read the file into a stream

    using (var stream = syncFileService.OpenRead(file))
    {
        if (stream != null)
        {
            mediaService.SetMediaFileContent(relativePath, stream);
            stream. Close();
        }
    }
    

    so in theory - none of the temp file stuff is touching Umbraco - Umbraco gets a memoryStream of the file, and asked to save it. and when we ask for the file, we are asking based on the value in UmbracoFile (e.g /media/folder/image.png )

    so if all that is right (and we haven't missed a bit somewhere) your file provider doesn't need to know we've done that any where. because we have read and written to a temp folder, but not passed that location to anyone ?

    Hope that makes sense ?

  • Dat Nguyen 7 posts 87 karma points
    Nov 08, 2023 @ 13:45
    Dat Nguyen
    0

    Hi,

    Thanks. It makes perfect sense.

    I added a breakpoint in MediaService.cs but it was never called on the receiver.

     public void SetMediaFileContent(string filepath, Stream stream)
     {
         _mediaFileManager.FileSystem.AddFile(filepath, stream, true);
     }
    

    I will do some more digging

  • Dat Nguyen 7 posts 87 karma points
    Nov 09, 2023 @ 10:27
    Dat Nguyen
    0

    Found the issue. This method GetPackMediaFIles is called on the receiver instance and assumes that the mediafiles are located in /media, however, in my case my mediafiles are placed under /umbraco/backoffice/api/mediafiles which makes sense why the mediafile is never uploaded to my SQL media store:

    public IEnumerable<string> GetPackMediaFiles(string folder)
    {
        string text = Path.Combine(folder, "media");
        if (!syncFileService.DirectoryExists(text))
        {
            return Enumerable.Empty<string>();
        }
        return from x in Directory.GetFiles(text, "*.*", SearchOption.AllDirectories)
            where !x.EndsWith("_names.json")
            orderby x
            select x;
    }
    

    Is it possible to get a optional configuration in appsettings where we can define a custompath for receveived mediefiles?

  • Kevin Jump 2342 posts 14889 karma points MVP 8x c-trib
    Nov 09, 2023 @ 10:46
    Kevin Jump
    100

    Hi Dat,

    I this instance that method is reading the media from the temp folder, where the media 'should' be in the media folder

    This is because the packs make the media path 'generic' and then set it back to any custom value on import .

    This is done so it can be portable between places where it is stored as /media and others where its something else

    have you got the UmbracoMediaPath value set in your config? if so, then that should be picked up and used.

    where the GetPackMediaFiles is called it gets a list of media and the path is later pushed through the GetLocalMediaPath method before being set on the media it.

        /// <summary>
        ///  get the non generic version of the path.
        /// </summary>
        private string GetLocalMediaPath(string source)
        {
            if (source.StartsWith(s_DefaultMediaPath) && !s_DefaultMediaPath.Equals(_mediaPath))
                return source.Replace(s_DefaultMediaPath, _mediaPath);
    
            return source;
        }
    

    in this instance _mediaPath is a trimmed version of the UmbracoMediaPath value (which defaults to "~/media"

    _mediaPath = globalSettings.CurrentValue.UmbracoMediaPath.Trim('~');
    
  • Dat Nguyen 7 posts 87 karma points
    Nov 09, 2023 @ 11:10
    Dat Nguyen
    0

    Thanks Kevin.

    This was the missing piece.

    Br, Dat

Please Sign in or register to post replies

Write your reply to:

Draft