Dear Publishing Partners,
As you know, over the past weekend, PressReader suffered from a major service interruption, the biggest one in this company’s 15-year history. We understand that this situation affected you and your teams negatively, and we sincerely apologize for that. To continue to be as transparent as possible, we want to share with you more details on what exactly happened and what we are doing to prevent this situation from happening again.
On Friday morning (UTC-7), we released the much-anticipated version of our new PressReader app for iOS. After closely monitoring its successful release, and since Saturday is our lowest usage day of the week, we then decided to release the new Android app.
We have performed many of these platform-wide updates in the past without any issues, so what was different with this one? You may remember that Google recently introduced an automatic app update mechanism which meant that when a developer releases an update to an app, the user gets the update without having to do anything. As an additional step, all update activations need to be registered in our backend to ensure the utmost security of your content (part of our DRM policy). This, therefore, meant that this time, instead of processing user-activated updates, millions of update requests were made to our servers all at the same time. Because of the many technical and security server checks in place, this created an Unintentional Denial of Service (UDoS), which inadvertently affected all of our services, PressReader and publisher-branded digital editions.
We reacted quickly and mobilized our development team who worked overnight to develop a new process to redirect user traffic from affected sources. The new setup allowed our servers to normalize the flow of update requests and process each one of them accordingly. We also released 2 new versions of our iOS app (4.0.1 and 4.0.2 – thanks to quick internal turnaround and Apple’s expedited review) to help alleviate the load. This process took about 12 hours to implement and by mid-morning Saturday (UTC-7), all our iOS, Windows 8, BlackBerry 10 and web operations were back to normal, albeit slower than usual.
Unfortunately for the Android users, since it is impossible to pull apps from Google Play once they have been published, additional development was needed to rectify the situation and it was only by Sunday end of day (UTC-7) that we were able to rectify the situation completely and to confirm that all services were back up and operational.
We take the recent events very seriously and since then we have been reviewing our upgrade protocols and are planning extensive expansion of cloud-based capacities to meet the ever-growing consumer demand. We have also separated our web and app traffic to limit their effect on one another, which will help in redirecting potentially affected users to other ways of getting the content. What this means is if by any chance one of the platforms is experiencing access issues, these will not affect other platforms and users will continue to have uninterrupted access to content.
We understand that although some updates were provided via our ND Connect Intranet, we have not been as effective at communicating progress as we should have been, especially on Saturday. We are therefore reviewing our communication procedures to ensure that you always receive the most up-to-date information regarding the status of our platforms. We will be sending you more updates on this soon, but in the meantime, ND Connect Intranet remains the best channel to get timely notifications and updates on the status of our network and platforms.
That being said, our industry is one that is changing quickly and continuously, and its development landscape is just as dynamic. Many of these changes we can predict, but others we will have to continue to adapt in an innovative manner. We can’t promise that the road ahead will always be a smooth one, but by continuing to be proactive and confront each situation at a time, PressReader is committed to building a network and platforms that will continue to successfully bring your content to your readers in the future.
Again, we sincerely apologize for any inconvenience this may have caused to you and your readers and we thank you for your understanding, trust and support. If you have any further questions at all, please don’t hesitate to contact our Content team directly.
Chief Content Officer and General Counsel