When Genevieve Bell's ATM machine wished her a happy birthday, she found it a bit creepy. When it wished her happy anniversary she was downright disturbed.
"It felt like they were tipping their hand a little too much," she said. "I had to take a step back and say 'ooh that's not good'. You've now told me much more about your tracking of me than I needed to know, in the guise of a personal experience. What you've done has gotten really creepy."
Speaking at the AIIA Navigating Digital Government Summit in Canberra today, the former Intel anthropologist warned that government and industry needed to keep in mind the effect data and technology had on people.
"It got too close. It was just too personal," the Australian National University professor said of her unnerving ATM experience.
While data can give rise to "magic" user interactions, Bell explained, it could also cause people to be anxious about what was known about them.
"We talk about big data and recommendation engines and algorithms. Consumers encounter it through preferences, they encounter it through things being served up to them which inevitably creates a sense of anxiety and questioning. Why is it that Netflix thinks I should watch that show? What is it they know about me that they imagine I should?" she said.
Humans can sometimes fear their choices are being "prescribed by their past" by these algorithms, which by their nature work on retrospective data.
"They want to imagine they're more interesting than that," she added.
What cows want
Making data visible and opening up analysis could also have unintended consequences, Bell warned, and data should not be considered a "benign thing".
Bell, who joined the ANU in February, recounted an Intel study into household water usage. In one case when a couple could see that one took a shower in the early evening after a business meeting, the other became suspicious of why.
"Making data visible has the capacity to make a series of tacit rituals public and visible. And it turns out that's a little bit confronting," she said.
Another study concerning agricultural IoT gave dairy cows RFID tags to they could trigger a automated milking process by themselves. While the farmers usually milked twice daily, given the ability, the cows were doing so far more often.
"Suddenly there's this moment: 'We need to think about what cows want'. And that was uncomfortable. Scale that up and imagine what data can tell us about what things want that we hadn't thought about."
Human in the middle
Bell, who is recognised as a world leader in the ethnographic approach to developing technology, explained that though it was easy to be wooed by "provocative and remarkable and delightful" innovations, human users can't be forgotten.
"If all we do is think about the technology we set ourselves up to not be as successful as we could and should be," she said. "Having humans in the middle, both as the objects and subjects and regulators of that technology is the most important and in some ways the hardest thing to do.
"It's not just talking about digital transformation but digital transformation in a world where we are still human beings."
@georgepnott
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.