The M365 Copilot, a digital assistant by Microsoft, was utilized in a three-month trial by the UK’s Department for Business and Trade to assess its impact on workplace productivity. During the trial from October to December 2024, 1,000 licenses were distributed, with a mix of volunteers and randomly selected staff using them. A study was subsequently conducted to evaluate time savings, quality assurance, and productivity gains.
The results of the trial revealed that while M365 Copilot was satisfactory to 72% of its users, the effect on productivity was mixed. Employees frequently used the tool, with two-thirds accessing it at least once a week, and 30% daily. The primary applications used were Word, Teams, and Outlook, while Loop and OneNote saw minimal engagement. When M365 Copilot was used, the tool’s primary applications involved tasks like transcribing meetings, writing emails, and summarizing communications, which users found quite satisfactory.
The actual efficiency of M365 Copilot demonstrated some discrepancies. Users were quicker at producing summaries and writing emails compared to non-users, though the time saved was minimal. Conversely, tasks involving Excel and PowerPoint, while completed faster, suffered in quality and accuracy. This necessitated additional corrective actions, undermining the purported time savings.
During interviews, some participants reported that the Copilot allowed them to handle routine administrative tasks more efficiently, which theoretically freed up time for strategic activities, training, or personal breaks. Yet, the productivity enhancements were not substantiated by the data, leading to a conclusion that increased efficiency in task completion did not translate directly into greater overall productivity.
Interestingly, 22% of participants reported experiencing “hallucinations” or errors in output by the M365 Copilot. This aspect of AI performance, where the system generates nonsensical or irrelevant output, reflects ongoing challenges with the technology’s reliability.
The study also highlighted dynamics within the department, noting that colleague and managerial attitudes significantly influenced the uptake and use of M365 Copilot. In some instances, teams were supportive, while in others, the introduction of AI tools was met with resistance.
This trial’s outcome casts doubt on the straightforward productivity benefits of M365 Copilot, aligning with wider skepticism about the practical value of generative AI in corporate settings. A recent MIT survey echoed similar sentiments, revealing that significant investments in generative AI by companies have yet to yield substantial returns.
Despite the mixed results, the trial provided insightful feedback on the potential and limitations of generative AI tools in enhancing workplace productivity. While M365 Copilot proved useful for routine and simple tasks, its effectiveness waned with more complex applications, requiring a nuanced understanding and adjustment of expectations for such technologies in the workplace. The Department’s ongoing analysis on the environmental impact and cost-effectiveness of M365 Copilot will further clarify the full scale of AI’s utility in corporate environments.
Read the full post on theregister.com
