Copied to clipboard

Flag this post as spam?

This post will be reported to the moderators as potential spam to be looked at


  • Joe Begly 14 posts 104 karma points
    Sep 27, 2021 @ 20:27
    Joe Begly
    0

    Umbraco Nine on Linux Server - Site Dies After Deploying Code Changes

    Hi everyone,

    I've gotten an Umbraco 9 instance up and running on a Linux server (Ubuntu 20.04). I'm using what appears to be the recommended approach of using Nginx as a reverse proxy to a Kestrel service

    https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-5.0

    The issue I'm having is that any time I do a code update and deploy it to my Linux server, followed by a restart of the kestrel service, the site dies. I get a 502 bad gateway error. Reloading the daemon does nothing. The only thing I've found that seems to work is if I navigate to place in the filesystem where I've deployed the published application and run the command:

    dotnet Umbraco.Nine.Linux.dll (Umbraco.Nine.Linux being the name of my project and the corresponding dll the result of the publish)

    that command just hangs so I hit ctrl + C to stop it, but that brings the site back up.

    Has anyone else experienced anything like this? If not, how are you handling code deployments to a Linux server? I should add that if I use the default .net core website from Visual Studio, I don't have this issue. I deploy my code and restart the kestrel service and everything is beautiful. So maybe there is some Umbraco startup process that's making things die?

    I've checked the logs for the kestrel service and they just indicate the service stops/restarts as expected. The logs in Nginx confuse me. They say something like

    2021/09/27 15:35:08 [error] 81048#81048: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 184.67.248.194, server: linux-net-core-umbraco-nine.jixaw.com, request: "GET /umbraco/api/test/another HTTP/1.1", upstream: "http://127.0.0.1:2003/umbraco/api/test/another", host: "linux-net-core-umbraco-nine.jixaw.com"
    

    I've also tried restarting nginx and that also seems to bring the site back up. Whether or not that's part of some rain dance, I don't know :-)

  • Joe Begly 14 posts 104 karma points
    Oct 04, 2021 @ 02:32
    Joe Begly
    0

    I'm still having this issue. Maybe a more focused question would be:

    Does anyone have Umbraco running on Linux and find that deploying updates goes smoothly? That is, no downtime? If so, what does your deployment strategy look like?

    I can deal with a slight lag like on Windows servers where the changes to the bin directory causes the site to sort of start over. End users don't get an error screen with that though. If I can't figure this out, I'm starting to think it's a deal breaker for me. I'll stay on Windows. Hosting on Linux was the most exciting thing about .net core though. Saving on hosting costs would be a big win.

    I haven't yet put this in a proper CI/CD pipeline yet. But if the act of deploying a zipped published site to the server, unzipping it to the place on the filesystem where the site lives and restarting the kestrel service makes the site die, I can't see how a scripted version in a CI/CD pipeline would behave differently. I'm either doing something just really naïve or I've got a bad version of Umbraco. As I mentioned above, this doesn't happen when I use a basic .net Core MVC sample site.

  • Joe Begly 14 posts 104 karma points
    Nov 03, 2021 @ 19:21
    Joe Begly
    0

    Crickets?

    Is anyone running an Umbraco 9 instance on Linux???

  • milkman matty 26 posts 108 karma points
    30 days ago
    milkman matty
    1

    I couldn't even get umbraco 9 to start on Linux, I'm waiting for it to be a bit more stable and sticking to Windows servers for the moment. It seems there is still quite a few issues in getting Umbraco to run in a Linux environment.

  • Joe Begly 14 posts 104 karma points
    30 days ago
    Joe Begly
    0

    I'm learning more as I research this further. It seems I'm probably expecting something I shouldn't be. I may need to look into containers, but I have a feeling I'll have issues there too.

    I'll share a few links as I collect my thoughts more and organize all this.

  • Carlos Mosqueda 221 posts 366 karma points
    29 days ago
    Carlos Mosqueda
    0

    @Joe, that would be very helpful.

    I have been looking at if Umbraco 9 can run reliably on Linux since hosting seems to be cheaper too. But Docker is looking like one of the better ways to go.

    If you find a way to run this on Linux or getting it to run in a Container, that would be huge. Warren Buckley did a video on how to get it running on a local IIS instance, but I have not seen anything on how to run it on Linux and only one tutorial on how to built it in a Docker container, but nothing on how to deploy it.

    Given my experience has always been with IIS up until now, I would love to see how to set up the reverse proxy and using Kestrel if it can be ran reliably that way. Nothing from HQ in the way of a tutorial on how to even do this. Not all of us totally understand the systems part of this.
    I don't host on my own machines at home, so also seeing how this is installed on an external hosting company would be hugely beneficial too.

  • Joe Begly 14 posts 104 karma points
    29 days ago
    Joe Begly
    0

    Sure. Like I mentioned, mine is running just fine on Linux. I don't mind detailing the steps I took to get to get it there...I did all this in AWS on an Ec2 instance running the latest Ubuntu and I've even hit it pretty hard with some load test scripts. I'm pretty sure a site would run if this was my production environment...just the fact that I don't know how to keep it from dying for 1 - 2 minutes while I'm deploying the code scares me a little lol!

    If containers are the way to go in order to get the code updates though, it may be a moot point.

    One thing I don't understand with the containers is how that would work with the media folder. I imagine that just getting "poofed" away each time I do a deployment, which won't work. Maybe they all need to go in s3 or something? I'm a bit of a novice with containers though so there's probably a very simple way to address this.

  • Joe Begly 14 posts 104 karma points
    28 days ago
    Joe Begly
    0

    @Carlos,

    Here is the one SO post I found that made me think I'm expecting something that's not possible:

    https://stackoverflow.com/questions/56869881/how-to-deploy-dotnet-core-code-on-linux-machine-without-stop-start-the-servcie

    The single response the topic got, from a very respected contributor, indicates that when deploying code changes to a netcore app, it needs to be stopped/started and the only way to avoid this is with containers and orchestrators.

    This one speaks about killing processes as something that would work. I can't imagine that being a smooth process and can't think how I'd even reliably script it (though I'm not very well versed in Linux).

    https://stackoverflow.com/questions/66112566/consolidating-deployment-steps-with-a-net-core-app-from-local-up-to-ec2-linux

    It's entirely possible that a better way to go is some sort of service like AWS's ElasticBeanstalk. But there is a cost associated with that. Will the time it takes me to figure out how to use a service like this (or even containers/orchestrators) eat up any savings I'd get with the cheaper hosting? Not sure.

    These are all unknowns for me though :-) I'd love to know I'm not barking up the wrong tree.

  • Joe Begly 14 posts 104 karma points
    1 week ago
    Joe Begly
    100

    For anyone who cares, after much bloodshed, I did get this working. I've abandoned the whole kestrel systemd thing and I'm using a docker container to run the app. After deploying code changes stopping/starting the container behaves much better than the kestrel service I was using before. If I hit the site at just the right time, yes, I still get the 502 bad gateway from nginx, BUT I can live with that because the site does recover very well. Anyone who won the lottery and hit a page and the instant I deploy an update could easily reload the browser and the site would be back up.

    To be more pedantic,

    I made a docker-compose.yml file that lives in my app root and it looks like this:

    version: '2.4'
    
    services:
      webapp:
        container_name: '<your_app_name>'
        #ex: 'my_awesome_app'
    
        image: mcr.microsoft.com/dotnet/aspnet:5.0
        restart: unless-stopped
    
        ###Pretty important
        #The first port listed is the host
        #This is the port on which nginx will be forwarding to us
        #The next port is fine to leave at 80 since docker will
        #use that without issue.
        #Because of this, we can ommit ASPNETCORE_URLS environment var
    
        ports:
          - "2005:80"
    
        #speaking of environment vars, we are loading them from a
        #file on the server so we can have app specific values
        #to things like connection strings
    
         env_file:
          - container-variables.env
    
        #here we tell docker to use our deployed (i.e. published)
        #code. This was important to me since I don't want to destroy
        #the media folder on deployments
    
        volumes:
          - <full/host/path/to/your/app>:/app
        #ex: /var/www/html/mysite:/app
    
        #this seems random.
    
        working_dir: /app
    
        #this way any files generated or media uploaded get run by the same user/group
        #on the web app directory. Otherwise, by default docker runs as 'root' and we don't want that
    
        user: <user>:<group>
        #ex: 1000:1000 for an Ubuntu ec2 instance in aws
    
        #whatever the name of your project's compiled dll is
    
        command: dotnet <Your.Built.Project.dll>
        #ex: MyStore.dll
    

    Hopefully those verbose comments help someone else out. Stuff like the port to use, the user and environment variables took some trial and error.

    Once that is done, in the app root, just do

    sudo docker-compose up -d
    

    and your site should run. So when it's time to deploy code, I upload the published file as a zip, unzip to the app location and run

    sudo docker-compose down
    sudo docker-compose up -d
    

    and everything is pretty happy. I've even got all this automated with an aws code pipeline so all I have to do is push to my git repository and my changes are up on the server in few minutes.

    I will say, for me, this is actually a huge step forward. Since we use AWS at my shop, I'd failed miserably trying to get something like this to work prior to Umbraco supporting .net core. AWS doesn't seem to have any docker images in their build servers that will successfully build a .net Framework app. They do support .net core.

    I've yet to battle test this (which I'll do soon with a home-brewed DDOS to make sure the site can handle reasonable traffic and the docker container doesn't puke). But it feels like it's more modern than hand copying files to a server like I've been doing for years on Windows machines.

Please Sign in or register to post replies

Write your reply to:

Draft