Towards estimating computer users’ mood from interaction behaviour with keyboard and mouse

Abstract

The purpose of this exploratory research was to study the relationship between the mood of computer users and their use of keyboard and mouse to examine the possibility of creating a generic or individualized mood measure. To examine this, a field study (n = 26) and a controlled study (n = 16) were conducted. In the field study, interaction data and self-reported mood measurements were collected during normal PC use over several days. In the controlled study, participants worked on a programming task while listening to high or low arousing background music. Besides subjective mood measurement, galvanic skin response (GSR) data was also collected. Results found no generic relationship between the interaction data and the mood data. However, the results of the studies found significant average correlations between mood measurement and personalized regression models based on keyboard and mouse interaction data. Together the results suggest that individualized mood prediction is possible from interaction behaviour with keyboard and mouse.

Keywords

keyboard / mouse / interaction / mood measure / computer users / programming

References

[1] Brave S, Nass C. Emotion in human-computer interaction. The humancomputer interaction handbook: fundamentals, evolving technologies and emerging applications, 2002, 81−96

[2] Plutchik R. Emotion: a psychoevolutionary synthesis. Harper & Row New York, 1980

[3] Picard R W. Affective Computing. MIT Press, 2000

[4] Zimmermann P, Guttormsen S, Danuser B, Gomez P. Affective computing—a rationale for measuring mood with mouse and keyboard. International Journal of Occupational Safety and Ergonomics, 2003, 9(4): 539−551

[5] Klein J T. Computer response to user frustration. PhD thesis, Massachusetts Institute of Technology, 1998

[6] Lazar J, Jones A, Hackley M, Shneiderman B. Severity and impact of computer user frustration: a comparison of student and workplace users. Interacting with Computers, 2006, 18(2): 187−207

[7] Ross J M, Zhang H. Structured programmers learning object-oriented programming: cognitive considerations. ACM SIGCHI Bulletin, 1997, 29(4): 93−99

[8] Khan I A, Brinkman W P, Hierons R M. Domoods affect programmers’ debug performance? Cognition, Technology & Work, 2011, 13(4): 245−258

[9] Lang P J. The emotion probe: studies of motivation and attention. American Psychologist, 1995, 50(5): 372−385

[10] Buchanan T, Johnson J A, Goldberg L R. Implementing a five-factor personality inventory for use on the internet. European Journal of Psychological Assessment, 2005, 21(2): 115−127

[11] Valstar M, Patras I, Pantic M. Facial action unit recognition using temporal templates. In: Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication. 2004, 253−258

[12] Gendolla G H. On the impact of mood on behavior: an integrative theory and a review. Review of General Psychology, 2000, 4(4): 378−408

[13] Forgas J P, Fiedler K. Us and them: mood effects on intergroup discrimination. Journal of Personality and Social Psychology, 1996, 70(1): 28−40

[14] Manucia G K, Baumann D J, Cialdini R B. Mood influences on helping: direct effects or side effects? Journal of Personality and Social Psychology, 1984, 46(2): 357−364

[15] Ottati V C, Isbell L M. Effects on mood during exposure to target information on subsequently reported judgments: an on-line model of misattribution and correction. Journal of Personality and Social Psychology, 1996, 71(1): 39−53

[16] Isen A M, Simmonds S F. The effect of feeling good on a helping task that is incompatible with good mood. Social Psychology, 1978, 346−349

[17] Isen A M, Geva N. The influence of positive affect on acceptable level of risk: the person with a large canoe has a large worry. Organizational Behavior and Human Decision Processes, 1987, 39(2): 145−154

[18] Nygren T E, Isen A M, Taylor P J, Dulin J. The influence of positive affect on the decision rule in risk situations: focus on outcome (and especially avoidance of loss) rather than probability. Organizational Behavior and Human Decision Processes, 1996, 66(1): 59−72

[19] Pham M T. Representativeness, relevance, and the use of feelings in decision making. Journal of Consumer Research, 1998, 25(2): 144−159

[20] Johnson N, Galata A, Hogg D. The acquisition and use of interaction behaviour models. In: Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 1998, 866−871

[21] Gavrila D, Davis L. Towards 3-d model-based tracking and recognition of human movement: a multi-view approach. In: Proceedings of the 1995 International Workshop on Automatic Face-and Gesture-Recognition. 1995, 272−277

[22] Lefter I, Rothkrantz L J, Wiggers P, Van Leeuwen D A. Emotion recognition from speech by combining databases and fusion of classifiers. In: Proceedings of the 13th International Conference on Text, Speech and Dialogue. 2010, 353−360

[23] Bobick A F, Davis J W. Action recognition using temporal templates. Motion-Based Recognition, 1997, 9: 125−146

[24] Sakurazawa S, Yoshida N, Munekata N, Omi A, Takeshima H, Koto H, Gentsu K, Kimura K, Kawamura K, Miyamoto M, Arima R, Mori T, SekIya T, Furukawa T, Hashimoto Y, Numata H, Akita J I, Tsukahara Y, Matsubara H. A computer game using galvanic skin response. In: Proceedings of the 2nd International Conference on Entertainment Computing, ICEC ’03. 2003, 1−3

[25] Boucsein W, Backs R W. Engineering psychophysiology as a discipline: historical and theoretical aspects. Engineering Psychophysiology. Issues and Applications, 2000, 3−30

[26] Chanel G, Ansari-Asl K, Pun T. Valence-arousal evaluation using physiological signals in an emotion recall paradigm. In: Proceedings of the 2007 IEEE International Conference on Systems, Man and Cybernetics. 2007, 2662−2667

[27] Boucsein W, Thum M. Design of work/rest schedules for computer work based on psychophysiological recovery measures. International Journal of Industrial Ergonomics, 1997, 20(1): 51−57

[28] Haider E, Luczak H, Rohmert W. Ergonomics investigations of workplaces in a police command-control centre equipped with tv displays. Applied Ergonomics, 1982, 13(3): 163−170

[29] Schleifer L M, Ley R. End-tidal PCO2 as an index of psychophysiological activity during VDT data-entry work and relaxation? Ergonomics, 1994, 37(2): 245−254

[30] Lisetti C L, Nasoz F. Maui: a multimodal affective user interface. In: Proceedings of the 10th ACM International Conference onMultimedia. 2002, 161−170

[31] Mähr W, Carlsson R, Fredriksson J, Maul O, Fjeld M. Tabletop interaction: research alert. In: Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles. 2006, 499−500

[32] Khanna P, Sasikumar M. Recognising emotions from keyboard stroke patterns. International Journal of Computer Applications, 2010, 11(9): 1−5

[33] Tsihrintzis G A, Virvou M, Alepis E, Stathopoulou I O. Towards improving visual-facial emotion recognition through use of complementary keyboard-stroke pattern information. In: Proceedings of the 5th International Conference on Information Technology: New Generations. 2008, 32−37

[34] Russell J. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39(6): 1161−1178

[35] Bradley M M, Lang P J. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 1994, 25(1): 49−59

[36] Mehrabian A. Basic Dimensions for A General Psychological Theory. OG&H Publishers, 2008

[37] Mehrabian A. Behavioural treatment and bio-behavioural assessment: computer applications. Oelgeschlager, Gunn & Hain, 1980

[38] Cohen J. Statistical power analysis. Current Directions in Psychological Science, 1992, 1(3): 98−101

[39] Card K, Moran T, Newell A. The Psychology of Human-Computer Inter action. Lawrence Erlbaum Publishers, 1983

[40] Borenstein M, Hedges L V, Higgins J P, Rothstein H R. Introduction to meta-analysis. John Wiley & Sons, 2009

[41] Wang H, Prendinger H, Igarashi T. Communicating emotions in online chat using physiological sensors and animated text. In: Proceedings of the 2004 Extended Abstracts on Human Factors in Computing Systems Conference. 2004, 1171−1174

[42] Shepherd P. Tools for transformation. 2001

[43] Albinoni T. Adagio in g minor for organ and strings, Perivale: England Warner Classics, 1996

[44] Thompson W F, Schellenberg E G, Husain G. Arousal, mood, and the mozart effect. Psychological Science, 2001, 12(3): 248−251

[45] Silvia P J, Abele A E. Can positive affect induce self-focused attention? methodological and measurement issues. Cognition & Emotion, 2002, 16(6): 845−853

[46] Moby H. On everything is wrong. New York: Elektra, 1995

[47] Tukey J. Exploratory data analysis. Addison-Wesley Press, 1977

[48] Baumgartner R, Ryner L, Somorjai R, Summers R. Exploratory data analysis reveals spatio-temporal structure of null FMRI data. In: Proceedings of the 2000 International Society for Magnetic Resonance in Medicine. 2000, 1717

[49] Gaillard A. Theoretical and methodological issues in psychophysiological research. Engineering Psychophysiology, 1998

[50] Harper R. Being human: Human-computer interaction in the year 2020. Microsoft Research, 2008

[51] Robison J, McQuiggan S, Lester J. Evaluating the consequences of affective feedback in intelligent tutoring systems. In: Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. 2009, 1−6

[52] Zhai J, Barreto A. Stress detection in computer users through noninvasive monitoring of physiological signals. Biomedical Sciences Instrumentation, 2006, 42: 495−500

[53] Pantic M, Rothkrantz L J. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 2003, 91(9): 1370−1390

[54] Chitu A G, Rothkrantz L J, Wiggers P, Wojdel J C. Comparison between different feature extraction techniques for audio-visual speech recognition. Journal on Multimodal User Interfaces, 2007, 1(1): 7−20

RIGHTS & PERMISSIONS

2013 Higher Education Press and Springer-Verlag Berlin Heidelberg