Wednesday, January 16, 2013

False positives suck

I've been holding out on writing about my recent sailing trip from the Galapagos to Panama City, because during the trip there was an occurrence that seriously rattled me, disturbed my crew-mates and generally marred a trip that had begun and promised to continue in a very chilled out atmosphere. I was dealing with the aftermath of that event until just now, when I got off the phone. Not wanting to tell an incomplete story, I suppose that now that that aftermath has concluded, I can go ahead.

On the second day of our passage my SPOT satellite messenger malfunctioned. I am frankly exhausted hearing about, re-telling, writing down and trying to intellectually and emotionally process what ensued, so I cannot bring myself to put it all down in detail one more time here. I'm sorry. I'm sure if I were a good journalist, it would spin an excellent yarn. And there are obviously a number of very very serious implications, so maybe we can talk about those in the comments, or on twitter, or something -- I'm pretty sure I know all of the 4 people who follow this blog personally -- because it's the implications that matter.

I had been using the SPOT the way I usually do, pressing OK once a day, to let my partner (via text & email) and my parents (via the website) know that all was well. I've used the SPOT this way for about 14 months, maybe 3-4 trips. Well, on New Year's Day, instead of transmitting an OK, the SPOT freaked out and started transmitting a spurious stream of "Help" and occasional "Cancelled" messages. Take a second to think about that. Deep breath. Continue. 25 minutes later, unaware that anything was amiss, I tried to turn off the device, as I always do, to save the batteries. It wouldn't turn off, so I thought, "wtf?", popped off the back and took out the batteries. That did it. That's when the random stream of "Help" messages stopped.

Several hours later, via a call from US Coast Guard to the boat's sat-phone -- which was not configured to take incoming phone calls, thereby blowing the skipper's mind -- we found out that my partner, my parents, the US, Ecuadorean and Greek Coast Guards and my private medevac insurance provider, Global Rescue, had all spent the previous few hours trying to ascertain my well being, looping in a half dozen additional minor players in the process. Had the USCG not magically been able to raise us on a sat-phone (one that has never in the past had and never in the future will have another incoming call -- magic!) the next step that was being proposed was to scramble a C130 spotter plane. Once the USCG reached us, the situation was put to sleep quickly and in an orderly fashion.

So, some distilled thoughts:

1) My partner and I have a protocol that we discuss in advance of each trip for how to respond to each of the possible SPOT messages (OK, Help, SOS and "custom", which we use as "cancel the protocol"). The protocol is basically the same in most use-cases (sailing on my boat, sailing on other boats and climbing), with small changes for what agencies to loop in. The protocol we had in place worked perfectly, so in hindsight we can look at this as a fire drill. Our agreed "Help" response is to wait two hours for an OK (thereby canceling the "Help") or an SOS (thereby immediately escalating it). That is, "Help" is what I'm supposed to push when there's something going wrong, and I am worried that I will not have a chance to press SOS later, but I'm working the situation at the moment, so there is no immediate need for assistance. E.g. the boat is heading for the rocks, but I'm trying to get an anchor down; or I've gone overboard, but I'm tethered and trying to get back on; or my partner's slid down a couloir, and I'm glissading down to check them out myself. If everything works out, I will press OK later. If things go to hell, I will press SOS. If I don't press either, two hours after the Help, my partner will assume that things have not gone well and that help is needed, and escalate to the SOS protocol. Even though in this case the "Help" message was sent as a result of hardware malfunction, there was no way my partner could have known, and there was nothing that he could or should have done differently. Our escalation strategy (first confirm the message with SPOT, then start looping in agencies and Global Rescue one by one, and follow their instructions) was correct. I would highly recommend that anyone who seriously uses SPOT plan their protocol in advance, together with the people who will have to enact it.

1b) I did learned something from a mistake I made. The moral: consider not having Help and SOS messages appear in your public mapping page. I normally don't have them appear, but I stupidly enabled them just for this trip, for no reason at all. The only person familiar with the abovementioned protocol was my partner, who was receiving messages directly, but my parents were also following along on the mapping page. Ideally, in case of Help and SOS, my partner should have been the only one to act. But through a time-zone vagary, my parents happened to look at the mapping page right after the spurious Help messages began. Thankfully, they contacted my partner first and he was able to more or less restrain them from taking actions outside the protocol. Mostly. The looping in of Greek Coast Guard was of their doing. In the end it proved invaluable in getting in touch with the UK Boat Registration Authority, through whom the sat-phone number was located and passed to US Coast Guard. But that's neither here nor there. I suppose my point is: choose whether you want just your protocol people, or the whole world to know about your Help and SOS messages, and set up your mapping page accordingly.

2) The agencies that my partner and parents contacted (US, Greek and Ecuadorean Coast Guard, Global Rescue) treated the SPOT "Help" message as credible. Nobody at any point suggested that the device had a track record of false positives or was unreliable in the least. That, at least, is heartening.

3) These agencies are AMAZING. They are total pros. They apparently did an incredible job sorting it all out, and interfacing with each other, and they were completely unfazed (the same cannot be said of some of the private parties involved) when it turned out to have been a malfunction. Treat these first responders with courtesy and respect, and they will save your hide. As first world citizens, we are so fortunate to have them available to back us up.

4) In the end, while the OK-mode for the SPOT is convenient and nice to have, I WILL NEVER USE IT AGAIN. I can understand how a device might fail to work, that is, how it can fail OFF. But I now know that an un-abused, un-wetted, comfortable SPOT can, for no apparent reason at all, fail ON. I cannot possibly take the chance that, while I'm just trying to say "Hi, I'm here, I'm OK", it will accidentally report me as being in distress, distressing all my people, in turn, and potentially launching a rescue. Screw the convenience and novelty of saying "I'm OK". The bottom line: I cannot do without SOS. I can do without OK. But then why own a SPOT instead of, say, a Personal Locator Beacon of some kind? But also, is the implication that people must personally experience a false positive before they realize that the risks associated with OK-mode are not worth it? And are false positives like this not bound to erode first responder confidence, in the long run? The answer to these questions depends on the specifics of the statistical distribution of false positives -- i.e. are they rare events, or are they significant (my data point: ~1/40 OKs turned into a false stream of Helps; that is *horrible*, statistically)? I hope, for the sake of those of us who may have to use a SPOT in a real emergency some day, that SPOT is doing its homework.

The final chapter in all this is SPOT's response. I wrote them a report of the incident, and said I wanted two things: a) for them to investigate the incident and learn something from it, and b) a replacement device, even though mine was 2 months out of warranty, and even though I vowed never to press OK again. Two days later a customer service agent called me who was obviously completely unaware of the implications of my story. She suggested that I pay $50 for a replacement device and initially had no comment on my sending it in for an investigation. I said that they could either send me a new device for free or cancel my account. She acquiesced and promised they would be sending me a warranty RMA email, with an address to which to mail the faulty device, and would be sending new a device. I pressed her on what kind of investigation they would conduct on the old device, but didn't get anything other than an assurance that they, in fact, would conduct one.

I think a chunk of the serenity prayer is relevant here...

Serene sunset, near the equator.