UK ISP PlusNet has issued an official ‘sorry’ after having lost 700Gb of their subscribers email and not succeeding recovered it.
The problem was initially reported on Sunday 9 July where it was reported that their “network engineers are currently investigating an issue where customers are not seeing any email displayed in their mailboxes.” We suspect that caused a collective sharp intake of breath at their offices.
PlusNet tell us that the problem took place during an upgrade of the email server. The engineer upgrading the servers NAS (Network Attached Storage) made the mistake of thinking that the mirrored drive that he was working on was the backup drive – it was in fact the live drive. By repartitioning the drive, this was then mirrored to the backup drive. Argh!
When they realised the problem, they stopped it double quick. By then the 700Gb of email had been lost. Within 3 hours they’d shipped the NAS to the data recovery specialist.
We feel sorry for the engineer who did this … we really do. They must have had many, many nightmares about this since and probably shake when they get close to a systems keyboard.
PlusNet say that their analysis shows that half of the lost mail was marked as spam, 48% of the mail had already been read, with only 1-2% not having been looked at. 1-2% doesn’t sound too much, but when it’s 700Gb of data that’s been lost, that is still 7Gb – 14Gb of loss.
To put this in perspective, the flow of email for any ISP is considerable with PlusNet getting 1.5 million ’email writes’ a day. They estimate that 700Gb of email takes about 3 days to build up. With this flow of data backing up to tape simply isn’t practical.
PlusNet proudly say on their home page that they ‘”top uSwitch.com customer satisfaction survey with 92%,” perhaps this will have dropped slightly – they said their customers have been “disappointed” – but given how open they’ve been during the process, we suspect their customer will at least know they’re honest – important in our view.
At the end of last year PlusNet bought MetroNet to add to their other ISPs.
Thanks to the story tip from Urban75