We recently identified a problem with our Catapult CMS Contact forms and Staff Directory Contact forms which may have affected your district. Specifically, emails submitted through Catapult websites failed to send between Monday, August 23 and Friday, August 27, 2021.
Once we identified this problem on Friday afternoon, 8/27, we corrected the underlying problem, allowing forms to return to normal operation. While no emails were lost during the week of 8/23, two fixes were required to send the backlog of unsent emails:
- Emails from Contact forms were recovered and started sending on Friday, 8/27 at 9pm.
- Emails from Staff Directory Contact forms need to be released individually by district and are slated to go out on Monday, 8/30.
In short, you may get a bundle of emails from Catapult all at once.
Underlying problem and solution:
We believe that the email email@example.com was shut down by its provider (Rackspace Business Email), because the system was overloaded with the volume of emails.
To avoid having this ever happen again, we have rebuilt the system to use Amazon Simple
Email Service (SES), a global leader in secure and reliable email service. This new build will be released Monday, 8/30, and any email that fails will try to resend every 20 seconds.
We realize this has come at a difficult time of year when you have a lot of things to worry about, and we are extremely sorry for any challenges this disruption has caused. We are confident that this new system will perform reliably for you.