First, allow me to apologize for to all of the
customers who received multiple copies of our email offer that went out
yesterday at approximately 3:00pm. We are seriously embarrassed that this
happened.
There were several technical issues in our mass-mailing process that
ultimately led to a subset of our customers receiving several copies of
the message. The proximate cause of the problem was that FogBugz
incorrectly interpreted an error message from our SMTP server, and
instead of stopping, it tried to resend the message several times.
Luckily, our awesome system administration team noticed the unusual
traffic and shut down the mail server, preventing an even bigger
problem.
The error that the mail server was throwing at FogBugz said that there
were too many recipients for a single message. There is a default
recipient cap in Postfix of 1000 addresses. Since each of the offer
emails messages were destined for over 1000 recipients, we were in
trouble. Ultimately, this configuration issue and the bug FogBugz had
with handling this particular error condition worked together to
confound us and unnecessarily send out extra emails to you.
We have begun the process of triaging the emails that did go out, and
rest assured that those folks who received a message from us will not
receive another copy. We will, starting immediately, be using a
different process for sending this kind of email. The bug that we found
in FogBugz has already been fixed, and that fix is being tested as I
write this.
Please accept our sincere apology for this inconvenience, and if there
is anything we can do for those affected, don't hesitate to contact us
at support@copilot.com.
