EZ NewsLetter for vBulletin 1.0.7-8

[Greg]

The new release of EZ NewsLetter adds support for sendmail piping services. With this the compatibility of EZ NewsLetter is expanded to most all email server setups available today.

We have also included a helper facility to aid you when a send batch fails and you have to start over on a page greater then page 1. The failure could be due to a failed internet connection or your computer power going out. The failure could be from trying to send too many emails per page.

In the first case, if the send failed and you know why, you will be able to just verify the email you are sending and continue with the existing per page setting.

If in either case you want to continue the mail with less email per page to slow things down a bit, you will be presented with suggestions on what number you should use as a per page setting to pick up where you left off properly.

That should make things a lot easier sorting out resends and getting used to the best settings for EZ NewsLetter on your server with the amount of email you need to send and the resources available.

[Dave A]

Hi Greg,

Been reading up with interest. I've got a couple of suggestions i.r.o. send rates.

1. Could you consider setting up a CRON job system (using the vB CRON method or linux based) for the send as an alternative to the automatic page load system you've got now.

2. Could you look at setting up a domain choke rate. The basic idea here is that you only send a limited number of emails to the same domain per minute (it's one of the features of phpList that helps make sure the mail gets through). I've been hit by this before - for some reason you can get people signing up from the same domain in batches, and having a domain choke definitely helps avoiding triggering spam filters.

[Greg]

A lot of people ask for cron. I have starting coding it. There are a lot of pitfalls. But we'll figure out something. I have put that on hold until everyone is satisfied that they can send mail. When I know we are done with the mail app itself, I can start a cron. I'm pretty sure we're done though with sending mail on 1.0.7.

Choking is going to be a pretty big sort. I'll have to think about how to pull it off. But yes, I'm on board. The exercise is in getting the mail through. So this is a great idea.

Back to the cron. To do cron from the vB cron for this type application is not a good idea. I'll say in advance we will not use the vBulletin cron. As I said, it's not easy. It's two fold, the vB cron will likely time out and there would be no indicators as to why or what happen. Pretty useless and a false sense of success.

Going to the server cron takes us out of the vB environment and causes us to ahve to do many many things by hand that vB would have done for us but can't because there is no user actually logged in.

So, two great ideas that are in the works. I do not have an ETA at all. Debugging the choking will be easy enough. But the cron is going to be an effort in patience to debug. I'll have to as I said, do many many things by hand vB would have done for us.

[Greg]

Here is a screen shot of what you will see in EZ NewsLetter when you change the start page on the first screen to a number greater than 1 to do a resend.

Often when you do a resend you want to reduce the number of emails per page. Doing the math can be daunting. So EZ NewsLetter will do it for you.

As you will see in this screen shot, when you click "Edit and Preview EZ NewsLetter TEXT and HTML Message Body" on the first screen, the next screen will prompt you with the numbers you need to do a resume and send from the proper next page with your new per page setting so you can resume at exactly after the last page sent.

If you are doing a resend and you know that the perpage setting is ok, you can leave it as is and continue with the existing per page setting already displayed in the input box.

[Greg]

Hi Greg,

2. Could you look at setting up a domain choke rate. The basic idea here is that you only send a limited number of emails to the same domain per minute (it's one of the features of phpList that helps make sure the mail gets through). I've been hit by this before - for some reason you can get people signing up from the same domain in batches, and having a domain choke definitely helps avoiding triggering spam filters.

We have the sort done. Now we need some metrics so we can apply it in the best case scenario.

We can sort the entire send in advance by domain to send to each domain in order. All the email to a.com would be first in line for example.

We can sort on a per send page basis too. Meaning the emails for the run of userids in order will be sorted to email domain order before sending.

Either way is ok, I'm not doing both. I think it would be best to figure out the best send rates and do it once in compliance with that however necessary. (I should be careful, I may be creating more work!)

So, can you share the sends rates you are aware of? I better find the RFC. But I'd like to hear your experience too.

Thanks,
Greg

[Dave A]

You work fast, Greg!
A lot of people ask for cron. I have starting coding it. There are a lot of pitfalls. But we'll figure out something.

To do cron from the vB cron for this type application is not a good idea. I'll say in advance we will not use the vBulletin cron.

Going to the server cron takes us out of the vB environment and causes us to ahve to do many many things by hand that vB would have done for us but can't because there is no user actually logged in.
How about setting an option to pump the send into the vB mail queue and use Paul M's Cron based E-mail sending (http://www.vbulletin.org/forum/showthread.php?t=167274). The only problem there is you get that infernal X-Mailer: vBulletin Mail via PHP line, even if you are mailing via SMTP! :(

On the server CRON (which would be first prize I think), php List uses a URL that includes log-in details. Is there a &username="username"&password="password" type functionality for vB? Maybe a clue lies in how the Google Adsense crawler handles logged-in only content?

Just thinking out loud - maybe something useful comes out of it.
So, can you share the sends rates you are aware of? I better find the RFC. But I'd like to hear your experience too.
That's tricky - some domains are fussier than others and hen you get a send-rate bounce, they don't tell you the limit.

I've had send rate bounces based on too many emails at once, too many emails in 5 minutes and too many emails in 1 hour.

As a guess, if the choke was set at a default of 1 email per domain per page/batch you are definitely safe. If you make it configurable, folks with big mail runs can push it up, but if a mail run takes 24 hours to complete and all the mail gets through, I'll take that ahead of getting send-rate bounces ;)

The curve ball in this is when you have a pile of websites using the same email server - I think the receiving server does the counts by the sender IP address or HELO and you won't see the fact that you're sending multiple emails to the same place unless you do a trace route in advance. I'm not suggesting trying to solve it, just tossing it in to increase understanding of what can go wrong with bulk mail.

[Greg]

We won't be using Paul's Code.

We won't be putting passwords in URLs. Wow, don't say that again please. :)

We need something definitive on the send rates. I'm digging through the RFC and even with that, each mail service has it's own ideas. MS for instance, like everything else, reinvented the wheel and is not standardized as the rest of the industry.

There is no throttle on php mail in vB for the most part. I'm sure forums with 20K members have mail going out constantly.

While I believe a send rate is a good idea, I do not believe it is the panacea of getting the mail through. I'll read more and collect the data to make a decision.


How many of you are using DNS SPF records?

How many of you have a send email, return to email and reply email in different domains? Those are factors too that raise flags. Using emails all in the same domain is a plus. Or use spf2 records.

Once I got my bounces under control, I stopped getting delayed 24, 48 and 72 hours delay messages. That's a sign of a flag, they tell your server no, wait. If you resend, you must be real, if you don't resend, they think it was spam.

And again, of course, it's all subjective. There is a standard, but almost no one implements it consistently and within the standard exactly. That's make for hard research and definitive compromise.

I'll keep digging. You keep me on my toes. :)

[Dave A]

And again, of course, it's all subjective. There is a standard, but almost no one implements it consistently and within the standard exactly. That's make for hard research and definitive compromise.
I've got to agree with that. It's been what has driven me nuts. Some of my local ISPs have big chunks of the market and have set really low tolerances to incoming send rates. To top it all, most of them are RFC ignorant.

Maybe I've got a unique regional problem :(

[Greg]

Nothing unique, you got it. I have to find the link, but the send rate per hour, or worse per half hour, from some ISPs is not going to serve almost anyone with a forum.

Using an ISP email is a bad idea with an Emailer like EZ NewsLetter. It's written for using domain email addresses and services we have control over. The sender is still responsible for send rates, we can't force anyone to use common sense. But the controls to do so are there and very useful.

[Dave A]

Here's one threshold that could serve as a useful choke:
A much better approach is to detect abusers by monitoring your outbound mailservers logs with automated scripts.
It is very unlikely that regular users will send email to more than 10 undeliverable recipients per hour.
Users sending email to multiple undeliverable addresses within a short time frame are almost always spammers, therefore you should shutdown those accounts automatically and promptly.
From UCEPROTECT here (http://www.uceprotect.net/en/index.php?m=4&s=0)
Monitor the bounces and suspend the send triggered by a bounce rate threshold.

[Greg]

That's a good link, Dave A. Thanks.

That's exactly my point. Senders have to get their bounces under control.

There are more than enough phycilities in vBulletin we can use as tools to opt the bounces out and then get the members motivated to fix their email addresses and opt back in.

I have not coded that part yet. We only have opt them out on bounce now. But rather than code it the way "I" would opt people out and notify them, I want to finish that part with the ideas of all of us that will need the tool.

So please, any ideas on the best way to opt members out and communicate to them they need to update their email and opt into email, let's hear them.

Thanks,
Greg

[Dave A]

So please, any ideas on the best way to opt members out and communicate to them they need to update their email and opt into email, let's hear them.
If there is a problem with their email address, you can't exactly send an email to the problem address to ask them to attend to the problem.

I've moved those members with this sort of problem into a banned group called "suspended." Then I set a notice via the Admin CP for this group saying their membership was suspended because of a problem - please contact me for details and to sort it out. It seems to work quite well, although I had to change a few phrases in vBulletin so as not to give them the idea they had been banned - rather suspended.

The point of setting it as a banned group is that vB then doesn't send any emails to the member, thereby eliminating the possibility of increasing your bounce count via an email address that you know is a problem.

Another idea for the future - test the email content against spam filters to get the spam score. My latest send had a link to a (perfectly legitimate) blog on blogspot, only to have bounces because Spamhaus is blocking any email with a link to blogspot.

[Greg]

Spam filters, while necessary, are self regulating agencies like those that report credit and crimial histories. They take no personal responsibility and are big beyond their necessity in idea.

I don't like the group change idea. I did it once and it happened to be an active member. They don't like no permission screens. We need a very soft tactful method of doing it.

[Greg]

Here is a list of things fixed and added to EZ NewsLetter 1.0.8 in addition to the things added in 1.0.7.

Fixed orders on holidays and birthdays bug.

Added code to fix changing the per page in the middle of a resend.
See screenshot above.

Fixed remove member form and added userid as an option to remove member or email address.

Added new publish news letter code. Make your newsletters available online with group permissions.

Selecting unregistered group makes published newsletters public to all.

SEO mod_rewrite URLs for online newsletters.
(Apache only. See sample.htaccess file.)

Add online newsletter to email option.

New settings for SEO Mode and online view.

New file forums/eznewsletter.php to display online newsletters.

EZ NewsLetter View Online with SEO mode URLs (http://www.cpurigs.com/forums/showthread.php?t=5209)

[Sonnie]

It just keeps getting better... looking good Greg!

[Dave A]

I don't like the group change idea. I did it once and it happened to be an active member.
If the person had visited in the last two weeks, I just sent them a PM rather than move them to the problem group.

I'd also changed the permissions of the problem group from the normal banned stuff.
They can read, but not post replies.
They can only PM and receive messages from admins.
And I made a number of changes to the no permission page to soften the blow.

BTW. I aim to buy b4 my next run in January - the exchange rate normally tends to strengthen in my favour during the holiday season. Although these are not normal times, there's not much risk in waiting.

[Greg]

What do ya'll think of a right column you can turn on and off and have different content in for each EZ NewsLetter? I bet you'll say no, left. No wait, both!

The only problem with per email settings is that they cause the first screen to grow immense it appears.

So what I'm thinking is that I'm going to break down the first page. We'll scale it back to the message and the settings. Then we will step through the process.

We could do the settings and messages. Then next the notices and optional CSS screen. Then the text and message previews.