First of all, nice package! :) This will be a great addition for a lot of people.
One question tho, can you tell us more about the way Newsletter Studio is handling the queueing of outgoing emails when there like thousands of subscribers on your list.
Hi Martijn! Well, Newsletter Studio saves all the subscribers in the different subscriptions. When a send out is started it will run on its own thread and you will be presented with a progress bar to track the progress.
In the background Newsletter Studio creates "tracking objects" to track each email. These objects will be updated with any error that may occur and if used, the bounce handler will update these "tracking objects" so that you can view all bounces or broken addresses.
The thread that are started will run in the background and if you where to close the dialog the dashboard will indicate that the newsletter send out is in progress. If you try to load the letter again the dialog will reappear.
The package has been tested with around 20 000 addresses and that work very well. The code that's running in the thread is very optimized, it's releasing all resources continually and won't cause any out of memory exceptions. Give it a spin on a local machine and let me know what you are thinking!
Thanks for your quick response. The only thing I am worried about is the speed of which emails are send out. Is this something you can control under settings? The reason I ask you this is because I dont want my mailserver getting blocked because of being identified as a 'spam mailserver'. Can you elaborate on this issue some more?
First of all this package may not be the best solution if you're trying to send newsletters to 1M subscribers every day. But if you're just doing a smaller send out (100K<) once a week or a month the SPAM-concerns should not be an issue.
I've been working with sites that's been doing daily mailings to over 50k of receivers without any problems so I would like to calm down all who are concerned.
About your question, there is no support for setting a pause or something between the sends but the thread releases everything and "starts over" after 20 "tracking objects" so that will give a natural pause of some 10-100 ms. If we find this a wanted feature there nothing that stops us from implementing it.
=D
If you don't agree on my conclusions in the text please guide me to some good reading.
Firstly, this looks like an excellent and much needed package, top work! Just to add my thoughts on the subject above...
I run an old mailing list system for a client which sends 70-80,000 emails per mail-out and typically runs once per month. I remember finding hosting for this system a few years ago and it was very difficult as every single host we approached (understandably) had a volume-per-time limit, usually undisclosed. However not many self-hosted mailing list systems supported throttling. My research indicated that if you tried to send many thousands of emails in a single batch through a typical hosting company's SMTP server, even if the process was nicely threaded so as not to time-out etc you would almost certainly come up against major problems with the sheer size of the queue and volume of mail being sent.
I think, if it's feasible for you to set up throttling limits, then it's going to open your software up to a much wider customer-base as it takes SMTP limitations - which are commonly very strict - out of the equation.
Anyhow, this is just my experience to add to the mix - I hope it helps.
So it looks like throttling limits, in other words, pause the send out for X seconds after Y emails has been sent, is something that a lot may want to have. And the ability to set up multiple SMTP-servers that the applications will switch between?
Jesper: Are you meaning that we would send the letters using there SMTPs or integrate into the API?
I mentioned it because of the state-of-the-art-keep-ip-out-of-blacklists-turbo-sending smtp server that can be used with your package. Maybe in a future version of Newsletter Studio you could consider making an optional tighter integration (push all receptients to sendgrid, push newsletter template and let them handle the stuff).
I will have that in mind but one of the main ideas is that you don't have to pay someone $49 per month just to send som newsletters. But I see what you mean that this would open up the product for more challenging scenarios. I'll have a look at this!
Thank you very much for all valuble input in this post! We are implementing both throttling and the ability to have multiple SmtpServers to iterate over but I would like to have your feedback on this. Right now there is one "over all setting" for throttling and for how many emails to send on each smtpserver. This confg-section will make it clearer:
My question is: Is this a good approach or should we be putting the setting on each individual server? One throttlig-setting for smtp1 and another for smtp2? Is this something that would be useful or is it overkill? From my research it seems that most other systems have a setup like this one.
In this post we have talked alot about throttling and multiple smtp-servers. Newsletter Studio now supports both. And if you want to use an external smtp-server there is alot of smtp relay-services out there, ie. Amazon has it's own.
What if, you send 50k e-mails, and suddenly your smtp-server stops or can't handle sending e-mails due to the large amount, or maybe due to other sites that are sending e-mails large amounts. Then your mail object possibly gets an error back, but will it retry to send the mails that got an error?
We now have a mailsystem that puts mails in a queue, and when the smtp is offline or it can't handle the large amount of mails. The queue will pause and resume when possible. Our websites access this mailsystem through a webservice, so our websites don't have smtp-settings.
My question is, how does newsletter studio react when a smtp-server is down or can't handle sending e-mails due to a large amount? (we can't setup an extra smtp-server to solve this problem)
What will happen at the moment is that the failure will be logged during the send out, no atempts to resend. We did not find a good way to know that the SMTP server is accualy down at the moment, just that it could not send the mail (reasones could be more than downtime ie. wrong email address, connectionproblems, recivers server is down etc).
Newsletter Studio will set a flag on all recivers that it could not send to (or that bounced later if you have that feature enabled). At the moment it will not try to send it again, but this could be something that we may concider in the future. Newsletter Studio dosen't run as a service so it has to relie on the send out thread that its creating.
I would love to know more about how you solved this issue in your que. How do you know that the server is down and that its not ie 50 email invalid email addresses that is bugging you?
There is a specific exception that will be thrown when an smtp server is down, or the server is too busy. Namely 'Too many connections...' exception. You can catch that exception, stop sending and try to resume your que when possible.
Yes, that could probably work, we got a new version comming in about a week, it will not include improvments reguarding this but there may be in the future. At the moment an SMTP-relay service with que-feature would be the best way to go. In the future we're looking at a feature that will make it possible for you to hand of the sent messages to some other service for send out. This will make is possible to use an SMTP-service, an API (or windows service) to perform the send out. But this is not yet on the roadmap and I cant tell you when it will be.
some technical details about sending e-mails
First of all, nice package! :)
This will be a great addition for a lot of people.
One question tho, can you tell us more about the way Newsletter Studio is handling the queueing of outgoing emails when there like thousands of subscribers on your list.
Thanks for your response.
Martijn
Hi Martijn!
Well, Newsletter Studio saves all the subscribers in the different subscriptions. When a send out is started it will run on its own thread and you will be presented with a progress bar to track the progress.
In the background Newsletter Studio creates "tracking objects" to track each email. These objects will be updated with any error that may occur and if used, the bounce handler will update these "tracking objects" so that you can view all bounces or broken addresses.
The thread that are started will run in the background and if you where to close the dialog the dashboard will indicate that the newsletter send out is in progress. If you try to load the letter again the dialog will reappear.
The package has been tested with around 20 000 addresses and that work very well. The code that's running in the thread is very optimized, it's releasing all resources continually and won't cause any out of memory exceptions.
Give it a spin on a local machine and let me know what you are thinking!
/ Markus Johansson
Enkel Media
Hi Markus,
Thanks for your quick response. The only thing I am worried about is the speed of which emails are send out. Is this something you can control under settings? The reason I ask you this is because I dont want my mailserver getting blocked because of being identified as a 'spam mailserver'. Can you elaborate on this issue some more?
Thanks :)
Martijn
Hi again!
First of all this package may not be the best solution if you're trying to send newsletters to 1M subscribers every day. But if you're just doing a smaller send out (100K<) once a week or a month the SPAM-concerns should not be an issue.
There's a lot of reading about this but there's no "exact answer". http://forums.asp.net/t/1724342.aspx/1
I've been working with sites that's been doing daily mailings to over 50k of receivers without any problems so I would like to calm down all who are concerned.
About your question, there is no support for setting a pause or something between the sends but the thread releases everything and "starts over" after 20 "tracking objects" so that will give a natural pause of some 10-100 ms. If we find this a wanted feature there nothing that stops us from implementing it.
=D
If you don't agree on my conclusions in the text please guide me to some good reading.
Hi Markus,
Firstly, this looks like an excellent and much needed package, top work! Just to add my thoughts on the subject above...
I run an old mailing list system for a client which sends 70-80,000 emails per mail-out and typically runs once per month. I remember finding hosting for this system a few years ago and it was very difficult as every single host we approached (understandably) had a volume-per-time limit, usually undisclosed. However not many self-hosted mailing list systems supported throttling. My research indicated that if you tried to send many thousands of emails in a single batch through a typical hosting company's SMTP server, even if the process was nicely threaded so as not to time-out etc you would almost certainly come up against major problems with the sheer size of the queue and volume of mail being sent.
I think, if it's feasible for you to set up throttling limits, then it's going to open your software up to a much wider customer-base as it takes SMTP limitations - which are commonly very strict - out of the equation.
Anyhow, this is just my experience to add to the mix - I hope it helps.
I would recommend using somthing like this: http://docs.sendgrid.com/documentation/get-started/integrate/
best
jesper
This is great input! Awesome!
So it looks like throttling limits, in other words, pause the send out for X seconds after Y emails has been sent, is something that a lot may want to have.
And the ability to set up multiple SMTP-servers that the applications will switch between?
Jesper: Are you meaning that we would send the letters using there SMTPs or integrate into the API?
Hi Markus,
I mentioned it because of the state-of-the-art-keep-ip-out-of-blacklists-turbo-sending smtp server that can be used with your package. Maybe in a future version of Newsletter Studio you could consider making an optional tighter integration (push all receptients to sendgrid, push newsletter template and let them handle the stuff).
I will have that in mind but one of the main ideas is that you don't have to pay someone $49 per month just to send som newsletters. But I see what you mean that this would open up the product for more challenging scenarios. I'll have a look at this!
Hi everyone!
Thank you very much for all valuble input in this post! We are implementing both throttling and the ability to have multiple SmtpServers to iterate over but I would like to have your feedback on this. Right now there is one "over all setting" for throttling and for how many emails to send on each smtpserver. This confg-section will make it clearer:
<outgoingEmail throttlingEnabled="false" throttlingMessagesPerBatch="50" throttlingSecondsBetweenBatches="60" messagesBeforeJumpingToNextServer="250">
<smtpServers>
<add name="Smtp1" host="smtp.example.com" port="21" username="" password="" enableSsl="false" />
<add name="Smtp2" host="smtp2.example.com" port="21" username="" password="" enableSsl="false" />
<add name="Smtp3" host="smtp3.example.com" port="21" username="" password="" enableSsl="false" />
</smtpServers>
</outgoingEmail>
My question is: Is this a good approach or should we be putting the setting on each individual server? One throttlig-setting for smtp1 and another for smtp2? Is this something that would be useful or is it overkill? From my research it seems that most other systems have a setup like this one.
Hi Markus,
I think it's enough to cover most/all - super.
best
jepsper
Thank you Jesper!
Yesterdays testing of the implementation was quite sexy. =D Anyone else who like the approach or have other ideas?
// Markus
I have yet to discover anything sexy about smtp :D
Hi everyone!
In this post we have talked alot about throttling and multiple smtp-servers. Newsletter Studio now supports both. And if you want to use an external smtp-server there is alot of smtp relay-services out there, ie. Amazon has it's own.
Please have a look at the new version, found here http://our.umbraco.org/projects/backoffice-extensions/newsletter-studio
What if, you send 50k e-mails, and suddenly your smtp-server stops or can't handle sending e-mails due to the large amount, or maybe due to other sites that are sending e-mails large amounts. Then your mail object possibly gets an error back, but will it retry to send the mails that got an error?
We now have a mailsystem that puts mails in a queue, and when the smtp is offline or it can't handle the large amount of mails. The queue will pause and resume when possible. Our websites access this mailsystem through a webservice, so our websites don't have smtp-settings.
My question is, how does newsletter studio react when a smtp-server is down or can't handle sending e-mails due to a large amount? (we can't setup an extra smtp-server to solve this problem)
Hi Michael!
What will happen at the moment is that the failure will be logged during the send out, no atempts to resend. We did not find a good way to know that the SMTP server is accualy down at the moment, just that it could not send the mail (reasones could be more than downtime ie. wrong email address, connectionproblems, recivers server is down etc).
Newsletter Studio will set a flag on all recivers that it could not send to (or that bounced later if you have that feature enabled). At the moment it will not try to send it again, but this could be something that we may concider in the future. Newsletter Studio dosen't run as a service so it has to relie on the send out thread that its creating.
I would love to know more about how you solved this issue in your que. How do you know that the server is down and that its not ie 50 email invalid email addresses that is bugging you?
There is a specific exception that will be thrown when an smtp server is down, or the server is too busy. Namely 'Too many connections...' exception. You can catch that exception, stop sending and try to resume your que when possible.
Hi Michael!
Yes, that could probably work, we got a new version comming in about a week, it will not include improvments reguarding this but there may be in the future. At the moment an SMTP-relay service with que-feature would be the best way to go. In the future we're looking at a feature that will make it possible for you to hand of the sent messages to some other service for send out. This will make is possible to use an SMTP-service, an API (or windows service) to perform the send out. But this is not yet on the roadmap and I cant tell you when it will be.
/ Markus
is working on a reply...