We know that we’ve caused a lot of frustration this month by limiting access to the payment reports. Here with an apology and an explanation is our Chief Technology Officer, Dom Lacey.
So Dom, tell us, what happened?
We noticed that we were starting to get some donation time-out alerts (where donations weren’t being processed because they were taking too long) due to the heavy traffic on the site.
We decided to suspend access to our payment reports temporarily, to create more space on our servers and make sure we could process all donations successfully.
Why did that happen?
The payment reports were pulling data from our database at the same time as the donation data was being written to the database. Due to the fact that more people were running more reports than normal, and the reports themselves were also bigger than normal, there were conflicts between the data that reports were accessing and the new data being created by donations.
So we had to decide whether to report on past donations and lose donations that were being made, or to process all donations and report on them later. We decided on the latter option, as our top priority is always making sure that people can donate to charities.
We have to admit that we didn’t plan ahead here. It hadn’t occurred to us that this year’s earlier marathon would coincide with heavier use of the charity account as charities tied up their year end accounts. It was poor planning from us. We’re really sorry and we’ll learn from it.
What are we doing to make sure it doesn’t happen again?
We’ve started work on three different projects to ensure this doesn’t happen next year.
1. We’ll pay more attention to the timing of the marathon and make sure reports are available during the end of March/early April.
2. We’ll separate the reporting database from our live database so there won’t be repeat of those data conflicts between reports and donations.
3. We’re re-thinking how we offer reports. We need to look how they are powered, generated and offered to charities. It may be that offering some reports in real time isn’t feasible, but we’re discussing ways of doing this to find the best solution for everyone.
Could you give us an analogy in layman’s terms?
OK, so imagining our database is the engine that drives the website, what we are planning to do is create a separate engine for reporting and – potentially – put it in a quicker, more reliable and more robust car!
Dom, thanks for this explanation. I know you and your team have worked extremely hard to make the best of a bad situation.
Yes, we tried to keep in contact with the charities team and to pass on updates as soon as there was any news.
We’re aware of the frustration we’ve caused to the charities we serve and we are genuinely sorry, as well as being determined to learn from our mistakes and engineer a better solution moving forward.
If you have any questions, please email me at jonathan@justgiving.com
