Reducing Customer Complaints through Algorithm-Generated Feedback: Evidence from a Field Experiment

Abstract

Leveraging algorithms to provide performance feedback to employees has become widespread in organizations. Algorithm-generated feedback is quite different from human’s feedback in feedback form and employees’ perceptions, so it is hard to directly predict the effect of algorithm-generated feedbacks. Despite the widespread use of algorithm-generated feedback in workplace, there is scant empirical evidence revealing its impacts. To address this gap, we empirically examine the effects of the implementation of an algorithm-generated feedback system through a field experiment conducted in the logistics industry. The results indicated that the algorithm-generated feedback significantly reduces customer complaints by about 20%. Additionally, employees with less work experience or lower workloads benefit from algorithm-generated feedback more. This work offers empirical evidence on the business value of algorithm-generated feedback and highlights the importance of employee characteristics in understanding and managing the effects of algorithmic supervision in the workplace.

Keywords

Algorithms / feedback / customer complaints / field experiment

References

[1] Abdul-Rahman S, Ali M F A M, Bakar A A, Mutalib S. Enhancing churn forecasting with sentiment analysis of steam reviews. Social Network Analysis and Mining, 2024, 14(1): 1-13.

[2] Ahn W K, Novick L R, Kim N S. Understanding behavior makes it more normal. Psychonomic Bulletin & Review, 2003, 10(3): 746-752.

[3] Allen R, Choudhury P. Algorithm-augmented work and domain experience: The countervailing forces of ability and aversion. Organization Science, 2022, 33(1): 149-169.

[4] Andrabi T, Das J, Ijaz Khwaja A I. Report cards: The impact of providing school and child test scores on educational markets. American Economic Review, 2017, 107(6): 1535-1563.

[5] Angrave D, Charlwood A, Kirkpatrick I, Lawrence M, Stuart M. HR and analytics: Why HR is set to fail the big data challenge. Human Resource Management Journal, 2016, 26(1): 1-11.

[6] Azmat G, Bagues M, Cabrales A, Iriberri N. What you don’t know can’t hurt you? A natural field experiment on relative performance feedback in higher education. Management Science, 2019, 65(8): 3714-3736.

[7] Bannert M. Managing cognitive loadrecent trends in cognitive load theory. Learning and Instruction, 2002, 12(1): 139-146.

[8] Barankay I. Rank incentives: Evidence from a randomized workplace experiment. Working Paper, 2012

[9] Baron R A. Criticism (informal negative feedback) as a source of perceived unfairness in organizations: Effects, mechanisms, and countermeasures. Justice in the Workplace: Approaching Fairness in Human Resource Management, 1993, USA. Lawrence Erlbaum Associates155-170

[10] Blackwell M, Iacus S, King G, Porro G. CEM: Coarsened exact matching in Stata. The Stata Journal, 2009, 9(4): 524-546.

[11] Blader S, Gartenberg C, Prat A. The contingent effect of management practices. The Review of Economic Studies, 2020, 87(2): 721-749. DOI:

[12] Bradley J V. Overconfidence in ignorant experts. Bulletin of the Psychonomic Society, 1981, 17(2): 82-84.

[13] Burnett J R, Lisk T C. The future of employee engagement: Real-time monitoring and digital tools for engaging a workforce. International Perspectives on Employee Engagement, 2021, USA. Routledge/Taylor & Francis Group117-128.

[14] Cabrera J M, Cid A. Gender differences to relative performance feedback: A field experiment in education. Documentos de Trabajo del Departamento de Economía, 2017

[15] Celik-Katreniak D. Dark side of incentives: Evidence from a randomized control trial in Uganda. SSRN: 3288474, 2018

[16] Chan T F. China is monitoring employees’ brain waves and emotions – and the technology boosted one company’s profits by $315 million. Business Insider, 2018

[17] Chen A. Employees say Google is trying to spy on them. That’ll be hard to prove. MIT Technology Review, 2019

[18] Choi E, Johnson D A, Moon K, Oah S. Effects of positive and negative feedback sequence on work performance and emotional responses. Journal of Organizational Behavior Management, 2018, 38(2–3): 97-115.

[19] Dahling J, O’Malley A L, Chau S L. Effects of feedback motives on inquiry and performance. Journal of Managerial Psychology, 2015, 30(2): 199-215.

[20] De Freitas J, Johnson S G. Optimality bias in moral judgment. Journal of Experimental Social Psychology, 2018, 79: 149-163.

[21] De Freitas J, Agarwal S, Schmitt B, Haslam N. Psychological factors underlying attitudes toward AI tools. Nature Human Behaviour, 2023, 7(11): 1845-1854.

[22] Desai V M. Learning to behave badly: Performance feedback and illegal organizational action. Industrial and Corporate Change, 2013, 23(5): 1327-1355.

[23] Drouvelis M, Paiardini P. Feedback quality and performance in organisations. The Leadership Quarterly, 2022, 33(6): 101534.

[24] Evans L, Kitchin R. A smart place to work? Big data systems, labour, control and modern retail stores. New Technology, Work and Employment, 2018, 33(1): 44-57.

[25] Fiedler F E. Cognitive resources and leadership performance. Applied Psychology, 1995, 44(1): 5-28.

[26] Fischer M, Wagner V. Effects of timing and reference frame of feedback: Evidence from a field experiment in secondary schools. Working Paper, 2017

[27] Gandini A. Labour process theory and the gig economy. Human Relations, 2019, 72(6): 1039-1056.

[28] Gent C. The politics of algorithmic management, 2018University of Warwick

[29] Goods C, Veen A, Barratt T. Is your gig any good? Analysing job quality in the Australian platform-based food-delivery sector. Journal of Industrial Relations, 2019, 61(4): 502-527.

[30] Gregory J B, Beck J W, Carr A E. Goals, feedback, and self-regulation: Control theory as a natural framework for executive coaching. Consulting Psychology Journal: Practice and Research, 2011, 63(1): 26.

[31] Gregory K. ‘My life is more valuable than this’: Understanding risk among on-demand food couriers in Edinburgh. Work, Employment and Society, 2021, 35(2): 316-331.

[32] Greve H R. Performance, aspirations, and risky organizational change. Administrative Science Quarterly, 1998, 44: 58-86. March)

[33] Greve H R, Gaba V. Performance feedback in organizations and groups: Common themes. Working Paper, 2017

[34] Heard J, Harriott C E, Adams J A. A survey of workload assessment algorithms. IEEE Transactions on Human-Machine Systems, 2018, 48(5): 434-451.

[35] Helberger N, Araujo T, de Vreese C H. Who is the fairest of them all? Public attitudes and expectations regarding automated decision-making. Computer Law & Security Review, 2020, 39: 105456.

[36] Horton E. PwC gives staff AI wristband to monitor stress levels during pandemic. Financial News, 2020

[37] Ilgen D R, Fisher C D, Taylor M S. Consequences of individual feedback on behavior in organizations. Journal of Applied Psychology, 1979, 64(4): 349.

[38] Ivanova M, Bronowicka J, Kocher E, Degner A. The app as a boss? Control and autonomy in application-based management. Working Paper, 2018

[39] Jussupow E, Benbasat I, Heinzl A. Why are we averse towards algorithms? A comprehensive literature review on algorithm aversion. Proceedings of the European Conference on Information Systems (ECIS 2020), 2020Morocco, June 15–17, 2020

[40] Jussupow E, Spohrer K, Heinzl A, Gawlitza J. Augmenting medical diagnosis decisions? An investigation into physicians’ decision-making process with artificial intelligence. Information Systems Research, 2021, 32(3): 713-735.

[41] Kajitani S, Morimoto K, Suzuki S. Information feedback in relative grading: Evidence from a field experiment. PLoS One, 2020, 15(4): e0231548.

[42] Kellogg K C, Valentine M, Christin A. Algorithms at work: The new contested terrain of control. Academy of Management Annals, 2020, 14(1): 366-410.

[43] Kim J S, Hamner W C. Effect of performance feedback and goal setting on productivity and satisfaction in an organizational setting. Journal of Applied Psychology, 1976, 61(1): 48.

[44] Kluger A N, DeNisi A. The effects of feedback interventions on performance: A historical review, a metaanalysis, and a preliminary feedback intervention theory. Psychological Bulletin, 1996, 119(2): 254.

[45] Kotiloglu S, Chen Y, Lechler T. Organizational responses to performance feedback: A meta-analytic review. Strategic Organization, 2021, 19(2): 285-311.

[46] Kotlar J, De Massis A, Fang H, et al. . Strategic reference points in family firms. Small Business Economics, 2014, 43(3): 597-619.

[47] Krenn B, Würth S, Hergovich A. The impact of feedback on goal setting and task performance: Testing the feedback intervention theory. Swiss Journal of Psychology, 2013, 72(2): 79-89.

[48] Leclercq-Vandelannoitte A. An ethical perspective on emerging forms of ubiquitous IT-based control. Journal of Business Ethics, 2017, 142(1): 139-154.

[49] Lee M K, Kusbit D, Metsky E, Dabbish L. Working with machines: The impact of algorithmic, data-driven management on human workers. Proceedings of the 33rd Annual ACM SIGCHI Conference, 2015Republic of Korea, April 18–23, 2015

[50] Leicht-Deobald U, Busch T, Schank C, Weibel A, Schafheitle S, Wildhaber I, Kasper G. The challenges of algorithm-based HR decision-making for personal integrity. Journal of Business Ethics, 2019, 160: 377-392.

[51] Levy K E. The contexts of control: Information, power, and truck-driving work. The Information Society, 2015, 31(2): 160-174.

[52] Levy K, Barocas S. Privacy at the margins: Refractive surveillance: Monitoring customers to manage workers. International Journal of Communication, 2018, 12: 23

[53] Lewellyn K B, Bao S R. R&D investment in the global paper products industry: A behavioral theory of the firm and national culture perspective. Journal of International Management, 2015, 21(1): 1-17.

[54] Li X, Hess T J, Valacich J S. Why do we trust new technology? A study of initial trust formation with organizational information systems. Journal of Strategic Information Systems, 2008, 17(1): 39-71.

[55] Liu X, Stoutenborough J, Vedlitz A. Bureaucratic expertise, overconfidence, and policy choice. Governance (Oxford), 2017, 30(4): 705-725.

[56] Locke E A. Social foundations of thought and action: A social-cognitive view. Academy of Management, 1987169-171

[57] Lunenburg F C. Goal-setting theory of motivation. International Journal of Management, Business, and Administration, 2011, 15(1): 1-6

[58] Maslach D. Change and persistence with failed technological innovation. Strategic Management Journal, 2016, 37(4): 714-723.

[59] Mateescu A, Nguyen A. Algorithmic management in the workplace, 2019

[60] Mateescu A, Nguyen A. Explainer: Workplace monitoring & surveillance, 2019

[61] McClelland M. I was a warehouse wage slave. The Best Business Writing 2013, 2013, US. Columbia University Press471-494.

[62] McKnight D H, Carter M, Thatcher J B, Clay P F. Trust in a specific technology: An investigation of its components and measures. ACM Transactions on Management lnformation Systems, 2011, 2(2): 1-25.

[63] Misztal B A. Normality and trust in Goffman’s theory of interaction order. Sociological Theory, 2001, 19(3): 312-324.

[64] Moore P, Robinson A. The quantified self: What counts in the neoliberal workplace. New Media & Society, 2016, 18(11): 2774-2792.

[65] Moore P V. The Quantified Self in Precarity: Work, Technology and What Counts, 2017, USA. Routledge.

[66] Muhr S L, Pedersen M, Alvesson M. Workload, aspiration, and fun: Problems of balancing self-exploitation and self-exploration in work life. Managing ‘Human Resources’ by Exploiting and Exploring People’s Potentials, 2012, UK. Emerald Group Publishing193-220.

[67] Murray D, Crilley G. The relationships between service problems and perceptions of service quality, satisfaction, and behavioral intentions of Australian public sports and leisure center customers. Journal of Park and Recreation Administration, 1999, 77(20): 42-64

[68] Orlikowski W J, Scott S V. What happens when evaluation goes online? Exploring apparatuses of valuation in the travel sector. Organization Science, 2014, 25(3): 868-891.

[69] Parent-Rocheleau X, Parker S K. Algorithms as work designers: How algorithmic management influences the design of jobs. Human Resource Management Review, 2022, 32(3): 100838.

[70] Park J A, Johnson D A, Moon K, Lee J. The interaction effects of frequency and specificity of feedback on work performance. Journal of Organizational Behavior Management, 2019, 39(3–4): 164-178.

[71] Pinheiro P G, Ramos J J, Donizete V L, Picanço P, De Oliveira G H. Workplace emotion monitoringan emotion-oriented system hidden behind a receptionist robot. Mechatronics and Robotics Engineering for Advanced and lntelligent Manufacturing, 2017, US. Springer International Publishing407-420.

[72] Plass J L, Moreno R, Brünken R. Cognitive Load Theory, 2010, UK. Cambridge University Press

[73] Rivera M, Qiu L, Kumar S, Petrucci T. Are traditional performance reviews outdated? An empirical analysis on continuous, real-time feedback in the workplace. Information Systems Research, 2021, 32(2): 517-540.

[74] Rosenblat A, Stark L. Algorithmic labor and information asymmetries: A case study of Uber’s drivers. International Journal of Communication, 2016, 10: 3758-3784

[75] Rosenblat A. Uberland: How Algorithms Are Rewriting the Rules of Work, 2018, Oakland. University of California Press.

[76] Ryberg J. Artificial intelligence at sentencing: When do algorithms perform well enough to replace humans?. AI and Ethics, 20241-10

[77] Schimmer M, Brauer M. Firm performance and aspiration levels as determinants of a firm’s strategic repositioning within strategic group structures. Strategic Organization, 2012, 10(4): 406-435.

[78] Schoeffer J, Machowski Y, Kuehl N (2021). A study on fairness and trust perceptions in automated decision making. arXiv: 2103.04757.

[79] Starke C, Baleis J, Keller B, Marcinkowski F (2022). Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature. Big Data & Society 9(2). Retrieved from https://doi.org/10.1177/20539517221115189.

[80] Tata J. The influence of managerial accounts on employees’ reactions to negative feedback. Group & Organization Management, 2002, 27(4): 480-503.

[81] Thorndike E L. Educational Psychology, Vol. 2. The Psychology of Learning, 1913, US. Legare Street Press.

[82] Vecchio R P. Theoretical and empirical examination of cognitive resource theory. Journal of Applied Psychology, 1990, 75(2): 141.

[83] Veen A, Barratt T, Goods C. Platform-Capital’s “App-etite” for control: A labour process analysis of food-delivery work in Australia. Work, Employment and Society, 2020, 34(3): 388-406.

[84] Villeval M C. Performance feedback and peer effects: A review. SSRN: 3543371, 2020

[85] Wood A J, Graham M, Lehdonvirta V, Hjorth I. Good gig, bad gig: Autonomy and algorithmic control in the global gig economy. Work, Employment and Society, 2019, 33(1): 56-75.

[86] Zheng X, Diaz I, Jing Y, Chiaburu D S. Positive and negative supervisor developmental feedback and task-performance. Leadership & Organization Development Journal, 2015, 36(2): 212-232.

RIGHTS & PERMISSIONS

Systems Engineering Society of China and Springer-Verlag GmbH Germany