Umbraco Nine on Linux Server - Site Dies After Deploying Code Changes
Hi everyone,
I've gotten an Umbraco 9 instance up and running on a Linux server (Ubuntu 20.04). I'm using what appears to be the recommended approach of using Nginx as a reverse proxy to a Kestrel service
The issue I'm having is that any time I do a code update and deploy it to my Linux server, followed by a restart of the kestrel service, the site dies. I get a 502 bad gateway error. Reloading the daemon does nothing. The only thing I've found that seems to work is if I navigate to place in the filesystem where I've deployed the published application and run the command:
dotnet Umbraco.Nine.Linux.dll
(Umbraco.Nine.Linux being the name of my project and the corresponding dll the result of the publish)
that command just hangs so I hit ctrl + C to stop it, but that brings the site back up.
Has anyone else experienced anything like this? If not, how are you handling code deployments to a Linux server? I should add that if I use the default .net core website from Visual Studio, I don't have this issue. I deploy my code and restart the kestrel service and everything is beautiful. So maybe there is some Umbraco startup process that's making things die?
I've checked the logs for the kestrel service and they just indicate the service stops/restarts as expected. The logs in Nginx confuse me. They say something like
I'm still having this issue. Maybe a more focused question would be:
Does anyone have Umbraco running on Linux and find that deploying updates goes smoothly? That is, no downtime? If so, what does your deployment strategy look like?
I can deal with a slight lag like on Windows servers where the changes to the bin directory causes the site to sort of start over. End users don't get an error screen with that though. If I can't figure this out, I'm starting to think it's a deal breaker for me. I'll stay on Windows. Hosting on Linux was the most exciting thing about .net core though. Saving on hosting costs would be a big win.
I haven't yet put this in a proper CI/CD pipeline yet. But if the act of deploying a zipped published site to the server, unzipping it to the place on the filesystem where the site lives and restarting the kestrel service makes the site die, I can't see how a scripted version in a CI/CD pipeline would behave differently. I'm either doing something just really naïve or I've got a bad version of Umbraco. As I mentioned above, this doesn't happen when I use a basic .net Core MVC sample site.
I couldn't even get umbraco 9 to start on Linux, I'm waiting for it to be a bit more stable and sticking to Windows servers for the moment. It seems there is still quite a few issues in getting Umbraco to run in a Linux environment.
I'm learning more as I research this further. It seems I'm probably expecting something I shouldn't be. I may need to look into containers, but I have a feeling I'll have issues there too.
I'll share a few links as I collect my thoughts more and organize all this.
I have been looking at if Umbraco 9 can run reliably on Linux since hosting seems to be cheaper too. But Docker is looking like one of the better ways to go.
If you find a way to run this on Linux or getting it to run in a Container, that would be huge. Warren Buckley did a video on how to get it running on a local IIS instance, but I have not seen anything on how to run it on Linux and only one tutorial on how to built it in a Docker container, but nothing on how to deploy it.
Given my experience has always been with IIS up until now, I would love to see how to set up the reverse proxy and using Kestrel if it can be ran reliably that way. Nothing from HQ in the way of a tutorial on how to even do this.
Not all of us totally understand the systems part of this.
I don't host on my own machines at home, so also seeing how this is installed on an external hosting company would be hugely beneficial too.
Sure. Like I mentioned, mine is running just fine on Linux. I don't mind detailing the steps I took to get to get it there...I did all this in AWS on an Ec2 instance running the latest Ubuntu and I've even hit it pretty hard with some load test scripts. I'm pretty sure a site would run if this was my production environment...just the fact that I don't know how to keep it from dying for 1 - 2 minutes while I'm deploying the code scares me a little lol!
If containers are the way to go in order to get the code updates though, it may be a moot point.
One thing I don't understand with the containers is how that would work with the media folder. I imagine that just getting "poofed" away each time I do a deployment, which won't work. Maybe they all need to go in s3 or something? I'm a bit of a novice with containers though so there's probably a very simple way to address this.
The single response the topic got, from a very respected contributor, indicates that when deploying code changes to a netcore app, it needs to be stopped/started and the only way to avoid this is with containers and orchestrators.
This one speaks about killing processes as something that would work. I can't imagine that being a smooth process and can't think how I'd even reliably script it (though I'm not very well versed in Linux).
It's entirely possible that a better way to go is some sort of service like AWS's ElasticBeanstalk. But there is a cost associated with that. Will the time it takes me to figure out how to use a service like this (or even containers/orchestrators) eat up any savings I'd get with the cheaper hosting? Not sure.
These are all unknowns for me though :-) I'd love to know I'm not barking up the wrong tree.
For anyone who cares, after much bloodshed, I did get this working. I've abandoned the whole kestrel systemd thing and I'm using a docker container to run the app. After deploying code changes stopping/starting the container behaves much better than the kestrel service I was using before. If I hit the site at just the right time, yes, I still get the 502 bad gateway from nginx, BUT I can live with that because the site does recover very well. Anyone who won the lottery and hit a page and the instant I deploy an update could easily reload the browser and the site would be back up.
To be more pedantic,
I made a docker-compose.yml file that lives in my app root and it looks like this:
version: '2.4'
services:
webapp:
container_name: '<your_app_name>'
#ex: 'my_awesome_app'
image: mcr.microsoft.com/dotnet/aspnet:5.0
restart: unless-stopped
###Pretty important
#The first port listed is the host
#This is the port on which nginx will be forwarding to us
#The next port is fine to leave at 80 since docker will
#use that without issue.
#Because of this, we can ommit ASPNETCORE_URLS environment var
ports:
- "2005:80"
#speaking of environment vars, we are loading them from a
#file on the server so we can have app specific values
#to things like connection strings
env_file:
- container-variables.env
#here we tell docker to use our deployed (i.e. published)
#code. This was important to me since I don't want to destroy
#the media folder on deployments
volumes:
- <full/host/path/to/your/app>:/app
#ex: /var/www/html/mysite:/app
#this seems random.
working_dir: /app
#this way any files generated or media uploaded get run by the same user/group
#on the web app directory. Otherwise, by default docker runs as 'root' and we don't want that
user: <user>:<group>
#ex: 1000:1000 for an Ubuntu ec2 instance in aws
#whatever the name of your project's compiled dll is
command: dotnet <Your.Built.Project.dll>
#ex: MyStore.dll
Hopefully those verbose comments help someone else out. Stuff like the port to use, the user and environment variables took some trial and error.
Once that is done, in the app root, just do
sudo docker-compose up -d
and your site should run. So when it's time to deploy code, I upload the published file as a zip, unzip to the app location and run
sudo docker-compose down
sudo docker-compose up -d
and everything is pretty happy. I've even got all this automated with an aws code pipeline so all I have to do is push to my git repository and my changes are up on the server in few minutes.
I will say, for me, this is actually a huge step forward. Since we use AWS at my shop, I'd failed miserably trying to get something like this to work prior to Umbraco supporting .net core. AWS doesn't seem to have any docker images in their build servers that will successfully build a .net Framework app. They do support .net core.
I've yet to battle test this (which I'll do soon with a home-brewed DDOS to make sure the site can handle reasonable traffic and the docker container doesn't puke). But it feels like it's more modern than hand copying files to a server like I've been doing for years on Windows machines.
Hello @Joe Begly,
I feel your trouble. I want to run UmbracoCMS as well on Linux and at the moment they don't have any good documentation on how to achieve the best uptime, simplest deployment.
I do it like this at the moment:
read -p "Where do you want to deploy? 1 for ArmbianC2, 2 for LinuxServer, 3 for AWS: " serverChoice
#echo $serverChoice
if [ $serverChoice == "1" ]
then
serverChoice="OdroidC2"
elif [ $serverChoice == "2" ]
then
serverChoice="LinuxServer"
elif [ $serverChoice == "3" ]
then
serverChoice="AWS"
fi
cd /Users/danieloberlechner/Projekte/2-CSharp/DanielOberlechner/
read -p "Is a full deploy needed? 1 for yes " fullDeploy
rm -rf ReleaseCandidateForServer
dotnet publish -c Release -o ReleaseCandidateForServer -f net7.0
if [ $fullDeploy == "1" ]
then
rsync -rv ReleaseCandidateForServer/ $serverChoice:/var/www/personalWebsite/
fi
rsync -rv appsettings* $serverChoice:/var/www/personalWebsite/
rsync -rv DanielOberlechner* $serverChoice:/var/www/personalWebsite/
rsync -rv umbraco/ $serverChoice:/var/www/personalWebsite/umbraco/
rsync -rv Views/ $serverChoice:/var/www/personalWebsite/Views/
rsync -rv wwwroot/ $serverChoice:/var/www/personalWebsite/wwwroot/
Then I created on my server a systemd service which looks like this:
So I hit my deploy script which can be in the future a CI/CD implementation. When the script has run threw everything is fine. If I made changes with C# I'll have to manually run this command:
systemctl restart personalWebsite
But this could also be achieved with CI/CD I guess.
Have you any hints for me if my deploy strategy is valid, or have you any hints how to make it better?
For Reverse Proxy I use at the moment caddy, but I would like to achieve everything also with Kestrel. I would like to skip reverse proxy because I have cloudflare for free in front of it ...
Have you any tips on how to run kestrel instead of nginx?
Am I the only one who thinks that the umbraco documentation should have a better deployment for linux lecture?
Interesting, I'm by no means a Linux expert so I can't comment on it too much.
Have you any tips on how to run kestrel instead of nginx? Am I the only one who thinks that the umbraco documentation should have a better deployment for linux lecture?
The part I'm confused about is how restarting the Kestrel service worked just fine for you. That was my initial roadblock with the whole thing. Seems to me you've figured out something I couldn't :-) What version of Umbraco are you running, and what Linux environment?
Hey,
I'm running Armbian 23.8.1 Jammy with Linux 6.1.50-current-meson64 locally at home and a AWS Amazon Linux variant in the cloud. Umbraco is the latest version 12.2.0 ... I started development with Umbraco short time ago and therefore always used version number 12 from umbraco cms.
One special thing I have in mind!!!
I'm not using Microsoft SQL Server but instead I use SQLite.
The reason behind this is I run on Linux and MS SQL should run rather on Windows and I have only a small site where SQLite runs just fine!
Thanks! This very well could be that the version I was using just had some inherent issues.
One special thing I have in mind!!! I'm not using Microsoft SQL Server but instead I use SQLite. The reason behind this is I run on Linux and MS SQL should run rather on Windows and I have only a small site where SQLite runs just fine!
That's interesting. I run all my sites on Linux and they connect to external databases that are Sql Server.
Umbraco Nine on Linux Server - Site Dies After Deploying Code Changes
Hi everyone,
I've gotten an Umbraco 9 instance up and running on a Linux server (Ubuntu 20.04). I'm using what appears to be the recommended approach of using Nginx as a reverse proxy to a Kestrel service
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-5.0
The issue I'm having is that any time I do a code update and deploy it to my Linux server, followed by a restart of the kestrel service, the site dies. I get a 502 bad gateway error. Reloading the daemon does nothing. The only thing I've found that seems to work is if I navigate to place in the filesystem where I've deployed the published application and run the command:
dotnet Umbraco.Nine.Linux.dll (Umbraco.Nine.Linux being the name of my project and the corresponding dll the result of the publish)
that command just hangs so I hit ctrl + C to stop it, but that brings the site back up.
Has anyone else experienced anything like this? If not, how are you handling code deployments to a Linux server? I should add that if I use the default .net core website from Visual Studio, I don't have this issue. I deploy my code and restart the kestrel service and everything is beautiful. So maybe there is some Umbraco startup process that's making things die?
I've checked the logs for the kestrel service and they just indicate the service stops/restarts as expected. The logs in Nginx confuse me. They say something like
I've also tried restarting nginx and that also seems to bring the site back up. Whether or not that's part of some rain dance, I don't know :-)
I'm still having this issue. Maybe a more focused question would be:
Does anyone have Umbraco running on Linux and find that deploying updates goes smoothly? That is, no downtime? If so, what does your deployment strategy look like?
I can deal with a slight lag like on Windows servers where the changes to the bin directory causes the site to sort of start over. End users don't get an error screen with that though. If I can't figure this out, I'm starting to think it's a deal breaker for me. I'll stay on Windows. Hosting on Linux was the most exciting thing about .net core though. Saving on hosting costs would be a big win.
I haven't yet put this in a proper CI/CD pipeline yet. But if the act of deploying a zipped published site to the server, unzipping it to the place on the filesystem where the site lives and restarting the kestrel service makes the site die, I can't see how a scripted version in a CI/CD pipeline would behave differently. I'm either doing something just really naïve or I've got a bad version of Umbraco. As I mentioned above, this doesn't happen when I use a basic .net Core MVC sample site.
Crickets?
Is anyone running an Umbraco 9 instance on Linux???
I couldn't even get umbraco 9 to start on Linux, I'm waiting for it to be a bit more stable and sticking to Windows servers for the moment. It seems there is still quite a few issues in getting Umbraco to run in a Linux environment.
I'm learning more as I research this further. It seems I'm probably expecting something I shouldn't be. I may need to look into containers, but I have a feeling I'll have issues there too.
I'll share a few links as I collect my thoughts more and organize all this.
@Joe, that would be very helpful.
I have been looking at if Umbraco 9 can run reliably on Linux since hosting seems to be cheaper too. But Docker is looking like one of the better ways to go.
If you find a way to run this on Linux or getting it to run in a Container, that would be huge. Warren Buckley did a video on how to get it running on a local IIS instance, but I have not seen anything on how to run it on Linux and only one tutorial on how to built it in a Docker container, but nothing on how to deploy it.
Given my experience has always been with IIS up until now, I would love to see how to set up the reverse proxy and using Kestrel if it can be ran reliably that way. Nothing from HQ in the way of a tutorial on how to even do this. Not all of us totally understand the systems part of this.
I don't host on my own machines at home, so also seeing how this is installed on an external hosting company would be hugely beneficial too.
Sure. Like I mentioned, mine is running just fine on Linux. I don't mind detailing the steps I took to get to get it there...I did all this in AWS on an Ec2 instance running the latest Ubuntu and I've even hit it pretty hard with some load test scripts. I'm pretty sure a site would run if this was my production environment...just the fact that I don't know how to keep it from dying for 1 - 2 minutes while I'm deploying the code scares me a little lol!
If containers are the way to go in order to get the code updates though, it may be a moot point.
One thing I don't understand with the containers is how that would work with the media folder. I imagine that just getting "poofed" away each time I do a deployment, which won't work. Maybe they all need to go in s3 or something? I'm a bit of a novice with containers though so there's probably a very simple way to address this.
@Carlos,
Here is the one SO post I found that made me think I'm expecting something that's not possible:
https://stackoverflow.com/questions/56869881/how-to-deploy-dotnet-core-code-on-linux-machine-without-stop-start-the-servcie
The single response the topic got, from a very respected contributor, indicates that when deploying code changes to a netcore app, it needs to be stopped/started and the only way to avoid this is with containers and orchestrators.
This one speaks about killing processes as something that would work. I can't imagine that being a smooth process and can't think how I'd even reliably script it (though I'm not very well versed in Linux).
https://stackoverflow.com/questions/66112566/consolidating-deployment-steps-with-a-net-core-app-from-local-up-to-ec2-linux
It's entirely possible that a better way to go is some sort of service like AWS's ElasticBeanstalk. But there is a cost associated with that. Will the time it takes me to figure out how to use a service like this (or even containers/orchestrators) eat up any savings I'd get with the cheaper hosting? Not sure.
These are all unknowns for me though :-) I'd love to know I'm not barking up the wrong tree.
For anyone who cares, after much bloodshed, I did get this working. I've abandoned the whole kestrel systemd thing and I'm using a docker container to run the app. After deploying code changes stopping/starting the container behaves much better than the kestrel service I was using before. If I hit the site at just the right time, yes, I still get the 502 bad gateway from nginx, BUT I can live with that because the site does recover very well. Anyone who won the lottery and hit a page and the instant I deploy an update could easily reload the browser and the site would be back up.
To be more pedantic,
I made a docker-compose.yml file that lives in my app root and it looks like this:
Hopefully those verbose comments help someone else out. Stuff like the port to use, the user and environment variables took some trial and error.
Once that is done, in the app root, just do
and your site should run. So when it's time to deploy code, I upload the published file as a zip, unzip to the app location and run
and everything is pretty happy. I've even got all this automated with an aws code pipeline so all I have to do is push to my git repository and my changes are up on the server in few minutes.
I will say, for me, this is actually a huge step forward. Since we use AWS at my shop, I'd failed miserably trying to get something like this to work prior to Umbraco supporting .net core. AWS doesn't seem to have any docker images in their build servers that will successfully build a .net Framework app. They do support .net core.
I've yet to battle test this (which I'll do soon with a home-brewed DDOS to make sure the site can handle reasonable traffic and the docker container doesn't puke). But it feels like it's more modern than hand copying files to a server like I've been doing for years on Windows machines.
Hello @Joe Begly, I feel your trouble. I want to run UmbracoCMS as well on Linux and at the moment they don't have any good documentation on how to achieve the best uptime, simplest deployment.
I do it like this at the moment:
Then I created on my server a systemd service which looks like this:
So I hit my deploy script which can be in the future a CI/CD implementation. When the script has run threw everything is fine. If I made changes with C# I'll have to manually run this command:
But this could also be achieved with CI/CD I guess.
Have you any hints for me if my deploy strategy is valid, or have you any hints how to make it better?
For Reverse Proxy I use at the moment caddy, but I would like to achieve everything also with Kestrel. I would like to skip reverse proxy because I have cloudflare for free in front of it ...
Have you any tips on how to run kestrel instead of nginx? Am I the only one who thinks that the umbraco documentation should have a better deployment for linux lecture?
Kind regards, Daniel
Interesting, I'm by no means a Linux expert so I can't comment on it too much.
The part I'm confused about is how restarting the Kestrel service worked just fine for you. That was my initial roadblock with the whole thing. Seems to me you've figured out something I couldn't :-) What version of Umbraco are you running, and what Linux environment?
Hey, I'm running Armbian 23.8.1 Jammy with Linux 6.1.50-current-meson64 locally at home and a AWS Amazon Linux variant in the cloud. Umbraco is the latest version 12.2.0 ... I started development with Umbraco short time ago and therefore always used version number 12 from umbraco cms.
One special thing I have in mind!!! I'm not using Microsoft SQL Server but instead I use SQLite. The reason behind this is I run on Linux and MS SQL should run rather on Windows and I have only a small site where SQLite runs just fine!
Hope could help you a bit further!
Greets, Daniel
Thanks! This very well could be that the version I was using just had some inherent issues.
That's interesting. I run all my sites on Linux and they connect to external databases that are Sql Server.
I must say I told fake news. Yesterday I updated my site again and nothing worked until i restarted services so my article before is trash.
I'm sorry but I thought it worked like a charme but it doesn't ...
My fault excuse me!
is working on a reply...