Reducing Customer Complaints through Algorithm-Generated Feedback: Evidence from a Field Experiment

Abstract

Leveraging algorithms to provide performance feedback to employees has become widespread in organizations. Algorithm-generated feedback is quite different from human’s feedback in feedback form and employees’ perceptions, so it is hard to directly predict the effect of algorithm-generated feedbacks. Despite the widespread use of algorithm-generated feedback in workplace, there is scant empirical evidence revealing its impacts. To address this gap, we empirically examine the effects of the implementation of an algorithm-generated feedback system through a field experiment conducted in the logistics industry. The results indicated that the algorithm-generated feedback significantly reduces customer complaints by about 20%. Additionally, employees with less work experience or lower workloads benefit from algorithm-generated feedback more. This work offers empirical evidence on the business value of algorithm-generated feedback and highlights the importance of employee characteristics in understanding and managing the effects of algorithmic supervision in the workplace.

Keywords

Algorithms / feedback / customer complaints / field experiment

References

[1] Abdul-RahmanS, AliM F A M, BakarA A, MutalibS. Enhancing churn forecasting with sentiment analysis of steam reviews. Social Network Analysis and Mining, 2024, 14(1): 1-13

[2] AhnW K, NovickL R, KimN S. Understanding behavior makes it more normal. Psychonomic Bulletin & Review, 2003, 10(3): 746-752

[3] AllenR, ChoudhuryP. Algorithm-augmented work and domain experience: The countervailing forces of ability and aversion. Organization Science, 2022, 33(1): 149-169

[4] AndrabiT, DasJ, Ijaz KhwajaA I. Report cards: The impact of providing school and child test scores on educational markets. American Economic Review, 2017, 107(6): 1535-1563

[5] AngraveD, CharlwoodA, KirkpatrickI, LawrenceM, StuartM. HR and analytics: Why HR is set to fail the big data challenge. Human Resource Management Journal, 2016, 26(1): 1-11

[6] AzmatG, BaguesM, CabralesA, IriberriN. What you don’t know can’t hurt you? A natural field experiment on relative performance feedback in higher education. Management Science, 2019, 65(8): 3714-3736

[7] BannertM. Managing cognitive loadrecent trends in cognitive load theory. Learning and Instruction, 2002, 12(1): 139-146

[8] BarankayI. Rank incentives: Evidence from a randomized workplace experiment. Working Paper, 2012

[9] BaronR A. Criticism (informal negative feedback) as a source of perceived unfairness in organizations: Effects, mechanisms, and countermeasures. Justice in the Workplace: Approaching Fairness in Human Resource Management, 1993, USA, Lawrence Erlbaum Associates: 155-170

[10] BlackwellM, IacusS, KingG, PorroG. CEM: Coarsened exact matching in Stata. The Stata Journal, 2009, 9(4): 524-546

[11] BladerS, GartenbergC, PratA. The contingent effect of management practices. The Review of Economic Studies, 2020, 87(2): 721-749

[12] BradleyJ V. Overconfidence in ignorant experts. Bulletin of the Psychonomic Society, 1981, 17(2): 82-84

[13] BurnettJ R, LiskT C. The future of employee engagement: Real-time monitoring and digital tools for engaging a workforce. International Perspectives on Employee Engagement, 2021, USA, Routledge/Taylor & Francis Group: 117-128

[14] CabreraJ M, CidA. Gender differences to relative performance feedback: A field experiment in education. Documentos de Trabajo del Departamento de Economía, 2017

[15] Celik-KatreniakD. Dark side of incentives: Evidence from a randomized control trial in Uganda. SSRN: 3288474, 2018

[16] ChanT F. China is monitoring employees’ brain waves and emotions – and the technology boosted one company’s profits by $315 million. Business Insider, 2018

[17] ChenA. Employees say Google is trying to spy on them. That’ll be hard to prove. MIT Technology Review, 2019

[18] ChoiE, JohnsonD A, MoonK, OahS. Effects of positive and negative feedback sequence on work performance and emotional responses. Journal of Organizational Behavior Management, 2018, 38(2–3): 97-115

[19] DahlingJ, O’MalleyA L, ChauS L. Effects of feedback motives on inquiry and performance. Journal of Managerial Psychology, 2015, 30(2): 199-215

[20] De FreitasJ, JohnsonS G. Optimality bias in moral judgment. Journal of Experimental Social Psychology, 2018, 79: 149-163

[21] De FreitasJ, AgarwalS, SchmittB, HaslamN. Psychological factors underlying attitudes toward AI tools. Nature Human Behaviour, 2023, 7(11): 1845-1854

[22] DesaiV M. Learning to behave badly: Performance feedback and illegal organizational action. Industrial and Corporate Change, 2013, 23(5): 1327-1355

[23] DrouvelisM, PaiardiniP. Feedback quality and performance in organisations. The Leadership Quarterly, 2022, 33(6): 101534

[24] EvansL, KitchinR. A smart place to work? Big data systems, labour, control and modern retail stores. New Technology, Work and Employment, 2018, 33(1): 44-57

[25] FiedlerF E. Cognitive resources and leadership performance. Applied Psychology, 1995, 44(1): 5-28

[26] FischerM, WagnerV. Effects of timing and reference frame of feedback: Evidence from a field experiment in secondary schools. Working Paper, 2017

[27] GandiniA. Labour process theory and the gig economy. Human Relations, 2019, 72(6): 1039-1056

[28] GentCThe politics of algorithmic management, 2018University of Warwick

[29] GoodsC, VeenA, BarrattT. Is your gig any good? Analysing job quality in the Australian platform-based food-delivery sector. Journal of Industrial Relations, 2019, 61(4): 502-527

[30] GregoryJ B, BeckJ W, CarrA E. Goals, feedback, and self-regulation: Control theory as a natural framework for executive coaching. Consulting Psychology Journal: Practice and Research, 2011, 63(1): 26

[31] GregoryK. ‘My life is more valuable than this’: Understanding risk among on-demand food couriers in Edinburgh. Work, Employment and Society, 2021, 35(2): 316-331

[32] GreveH R. Performance, aspirations, and risky organizational change. Administrative Science Quarterly, 1998, 44: 58-86 March)

[33] GreveH R, GabaV. Performance feedback in organizations and groups: Common themes. Working Paper, 2017

[34] HeardJ, HarriottC E, AdamsJ A. A survey of workload assessment algorithms. IEEE Transactions on Human-Machine Systems, 2018, 48(5): 434-451

[35] HelbergerN, AraujoT, de VreeseC H. Who is the fairest of them all? Public attitudes and expectations regarding automated decision-making. Computer Law & Security Review, 2020, 39: 105456

[36] HortonE. PwC gives staff AI wristband to monitor stress levels during pandemic. Financial News, 2020

[37] IlgenD R, FisherC D, TaylorM S. Consequences of individual feedback on behavior in organizations. Journal of Applied Psychology, 1979, 64(4): 349

[38] IvanovaM, BronowickaJ, KocherE, DegnerA. The app as a boss? Control and autonomy in application-based management. Working Paper, 2018

[39] JussupowE, BenbasatI, HeinzlA. Why are we averse towards algorithms? A comprehensive literature review on algorithm aversion. Proceedings of the European Conference on Information Systems (ECIS 2020), 2020Morocco, June 15–17, 2020

[40] JussupowE, SpohrerK, HeinzlA, GawlitzaJ. Augmenting medical diagnosis decisions? An investigation into physicians’ decision-making process with artificial intelligence. Information Systems Research, 2021, 32(3): 713-735

[41] KajitaniS, MorimotoK, SuzukiS. Information feedback in relative grading: Evidence from a field experiment. PLoS One, 2020, 15(4): e0231548

[42] KelloggK C, ValentineM, ChristinA. Algorithms at work: The new contested terrain of control. Academy of Management Annals, 2020, 14(1): 366-410

[43] KimJ S, HamnerW C. Effect of performance feedback and goal setting on productivity and satisfaction in an organizational setting. Journal of Applied Psychology, 1976, 61(1): 48

[44] KlugerA N, DeNisiA. The effects of feedback interventions on performance: A historical review, a metaanalysis, and a preliminary feedback intervention theory. Psychological Bulletin, 1996, 119(2): 254

[45] KotilogluS, ChenY, LechlerT. Organizational responses to performance feedback: A meta-analytic review. Strategic Organization, 2021, 19(2): 285-311

[46] KotlarJ, De MassisA, FangH, et al.. Strategic reference points in family firms. Small Business Economics, 2014, 43(3): 597-619

[47] KrennB, WürthS, HergovichA. The impact of feedback on goal setting and task performance: Testing the feedback intervention theory. Swiss Journal of Psychology, 2013, 72(2): 79-89

[48] Leclercq-VandelannoitteA. An ethical perspective on emerging forms of ubiquitous IT-based control. Journal of Business Ethics, 2017, 142(1): 139-154

[49] LeeM K, KusbitD, MetskyE, DabbishL. Working with machines: The impact of algorithmic, data-driven management on human workers. Proceedings of the 33rd Annual ACM SIGCHI Conference, 2015Republic of Korea, April 18–23, 2015

[50] Leicht-DeobaldU, BuschT, SchankC, WeibelA, SchafheitleS, WildhaberI, KasperG. The challenges of algorithm-based HR decision-making for personal integrity. Journal of Business Ethics, 2019, 160: 377-392

[51] LevyK E. The contexts of control: Information, power, and truck-driving work. The Information Society, 2015, 31(2): 160-174

[52] LevyK, BarocasS. Privacy at the margins: Refractive surveillance: Monitoring customers to manage workers. International Journal of Communication, 2018, 12: 23

[53] LewellynK B, BaoS R. R&D investment in the global paper products industry: A behavioral theory of the firm and national culture perspective. Journal of International Management, 2015, 21(1): 1-17

[54] LiX, HessT J, ValacichJ S. Why do we trust new technology? A study of initial trust formation with organizational information systems. Journal of Strategic Information Systems, 2008, 17(1): 39-71

[55] LiuX, StoutenboroughJ, VedlitzA. Bureaucratic expertise, overconfidence, and policy choice. Governance (Oxford), 2017, 30(4): 705-725

[56] LockeE A. Social foundations of thought and action: A social-cognitive view. Academy of Management, 1987169-171

[57] LunenburgF C. Goal-setting theory of motivation. International Journal of Management, Business, and Administration, 2011, 15(1): 1-6

[58] MaslachD. Change and persistence with failed technological innovation. Strategic Management Journal, 2016, 37(4): 714-723

[59] MateescuA, NguyenAAlgorithmic management in the workplace, 2019

[60] MateescuA, NguyenAExplainer: Workplace monitoring & surveillance, 2019

[61] McClellandM. I was a warehouse wage slave. The Best Business Writing 2013, 2013, US, Columbia University Press: 471-494

[62] McKnightD H, CarterM, ThatcherJ B, ClayP F. Trust in a specific technology: An investigation of its components and measures. ACM Transactions on Management lnformation Systems, 2011, 2(2): 1-25

[63] MisztalB A. Normality and trust in Goffman’s theory of interaction order. Sociological Theory, 2001, 19(3): 312-324

[64] MooreP, RobinsonA. The quantified self: What counts in the neoliberal workplace. New Media & Society, 2016, 18(11): 2774-2792

[65] MooreP VThe Quantified Self in Precarity: Work, Technology and What Counts, 2017, USA, Routledge

[66] MuhrS L, PedersenM, AlvessonM. Workload, aspiration, and fun: Problems of balancing self-exploitation and self-exploration in work life. Managing ‘Human Resources’ by Exploiting and Exploring People’s Potentials, 2012, UK, Emerald Group Publishing: 193-220

[67] MurrayD, CrilleyG. The relationships between service problems and perceptions of service quality, satisfaction, and behavioral intentions of Australian public sports and leisure center customers. Journal of Park and Recreation Administration, 1999, 77(20): 42-64

[68] OrlikowskiW J, ScottS V. What happens when evaluation goes online? Exploring apparatuses of valuation in the travel sector. Organization Science, 2014, 25(3): 868-891

[69] Parent-RocheleauX, ParkerS K. Algorithms as work designers: How algorithmic management influences the design of jobs. Human Resource Management Review, 2022, 32(3): 100838

[70] ParkJ A, JohnsonD A, MoonK, LeeJ. The interaction effects of frequency and specificity of feedback on work performance. Journal of Organizational Behavior Management, 2019, 39(3–4): 164-178

[71] PinheiroP G, RamosJ J, DonizeteV L, PicançoP, De OliveiraG H. Workplace emotion monitoringan emotion-oriented system hidden behind a receptionist robot. Mechatronics and Robotics Engineering for Advanced and lntelligent Manufacturing, 2017, US, Springer International Publishing: 407-420

[72] PlassJ L, MorenoR, BrünkenRCognitive Load Theory, 2010, UK, Cambridge University Press

[73] RiveraM, QiuL, KumarS, PetrucciT. Are traditional performance reviews outdated? An empirical analysis on continuous, real-time feedback in the workplace. Information Systems Research, 2021, 32(2): 517-540

[74] RosenblatA, StarkL. Algorithmic labor and information asymmetries: A case study of Uber’s drivers. International Journal of Communication, 2016, 10: 3758-3784

[75] RosenblatAUberland: How Algorithms Are Rewriting the Rules of Work, 2018, Oakland, University of California Press

[76] RybergJ. Artificial intelligence at sentencing: When do algorithms perform well enough to replace humans?. AI and Ethics, 20241-10

[77] SchimmerM, BrauerM. Firm performance and aspiration levels as determinants of a firm’s strategic repositioning within strategic group structures. Strategic Organization, 2012, 10(4): 406-435

[78] Schoeffer J, Machowski Y, Kuehl N (2021). A study on fairness and trust perceptions in automated decision making. arXiv: 2103.04757.

[79] Starke C, Baleis J, Keller B, Marcinkowski F (2022). Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature. Big Data & Society 9(2). Retrieved from https://doi.org/10.1177/20539517221115189.

[80] TataJ. The influence of managerial accounts on employees’ reactions to negative feedback. Group & Organization Management, 2002, 27(4): 480-503

[81] ThorndikeE LEducational Psychology, Vol. 2. The Psychology of Learning, 1913, US, Legare Street Press

[82] VecchioR P. Theoretical and empirical examination of cognitive resource theory. Journal of Applied Psychology, 1990, 75(2): 141

[83] VeenA, BarrattT, GoodsC. Platform-Capital’s “App-etite” for control: A labour process analysis of food-delivery work in Australia. Work, Employment and Society, 2020, 34(3): 388-406

[84] VillevalM C. Performance feedback and peer effects: A review. SSRN: 3543371, 2020

[85] WoodA J, GrahamM, LehdonvirtaV, HjorthI. Good gig, bad gig: Autonomy and algorithmic control in the global gig economy. Work, Employment and Society, 2019, 33(1): 56-75

[86] ZhengX, DiazI, JingY, ChiaburuD S. Positive and negative supervisor developmental feedback and task-performance. Leadership & Organization Development Journal, 2015, 36(2): 212-232

RIGHTS & PERMISSIONS

Systems Engineering Society of China and Springer-Verlag GmbH Germany