Surveys have been used for more than 150 years to get data about all kinds of opinions, attitudes, but also behaviors.
Usually, a standardized questionnaire is sent to a sample of the population of interest, and the answers are used to draw inferences about these opinions attitudes or behaviors within the complete population of interest. These inferences are often the basis for social, political or economic decisions.
But are surveys an adequate instrument to measure this whole spectrum of information?
It is often considered that surveys are useful to measure opinions and attitudes. However, the use of surveys in order to measure behaviors has been criticized, mainly because of the human memory limitations. People cannot remember everything.
In addition, what a person remembers can differ from the true fact. For instance, experiments about pain show that people remember a more painful experience if the pain is higher at the very end of the experience than if it is before, even if a similar level of pain is supported overall (Kahneman and Riis, 2005). Other experiments show that memories can be changed by things that we are told (Loftus and Ketcham, 1994).
For example, victims of physical violence may identify their aggressor wrongly if the person that interviews them gives clues in a wrong direction.
Furthermore, the problem of memory limitations has probably grown together with the development of new technologies. On the one hand, the digitalization of our lives has led to a reduced duration of events (for instance, you can buy a product in just a few clicks), a loss of context (you do not need to go the shop anymore), and an increased distraction (reception of emails or messages at any time).
On the other hand, the need for recall has been drastically reduced: we do not need to remember telephone numbers, Facebook tells us whose birthday it is today, etc.
In order to study the impact of this problem of memory in the frame of online surveys, we asked a sample of people from the Netquest online panel in Spain in May-June 2015 what sounds like a simple question: what are the last 5 websites that you visited using a given computer?
Then, we compared the answers with data coming from an external source: a tracker, installed on the given computer, which registers all the URLs of the webpages visited by the participants once they have agreed to install it. We considered that we had a match if the same website appeared in both sources, independently of the exact position. Thus, if a participant said “Google” in position 3 in the survey and we observed “Google” in position 5 in the tracking data, this is counted as a match.
Out of the 178 participants considered, only 1.7% gave the five same websites in the survey as the ones observed in the tracking data, only 6.2% gave four out of the five, and only 9.0% gave three out of five. No matches at all were observed for 27.5% of the participants.
These results clearly show that survey answers do not fit well what we observe using a tracker, suggesting that using surveys in order to study behaviors, in particular online behaviors, might lead to wrong conclusions.
So what should we do? The solution to this problem probably lies with the combination of different sources of data collection.
The survey is one of these sources; a source used for many years, that can still provide useful information, particularly in the field of opinion data. However, surveys also have limits, so complementing them with other data sources seems necessary.
In particular, when studying the online behaviors of people, new technologies provide now interesting tools to complement survey data with passive data, collected automatically when people surf the web: information about the URLs, as used in this study, but also information about online payment, social media usage, etc.
By combining more than one source of data collection a more global and accurate picture of the reality will be revealed.
Co-author: Carlos Ochoa.
Citation: Revilla, M., Ochoa, C., and G. Loewe (2016). Using passive data from a meter to complement survey data in order to study online behavior. Social Science Computer Review. doi:10.1177/0894439316638457.
Kahneman, D., and J. Riis (2005). “Living, and thinking about it: Two perspectives on life”. In F. Huppert, B. Keverne, & N. Baylis (Eds.), The science of well-being. Oxford, England: Oxford University Press.
Loftus, E.F. and K. Ketcham (1994). The Myth of Repressed Memory. NY: St. Martin’s Press.
Figure legend: This Knowridge.com image is for illustrative purposes only.
