Organisation type
Chambers of commerce
The problem
The assistant groups member responses by the dominant theme and the summary says members support the proposal. The minority objection disappears, themes nobody raised appear, and a consensus that does not exist is presented as the voice of the membership.
27 courses · 2 specific to this profession
A process from your work
How long does it take you to write a report from a member survey?
- Step 1From the results table, the tool drafts the findings and the paragraphs for the bulletin.
- Step 2For every percentage in the draft, you find its numerator and denominator in the table.
- Step 3The person responsible decides what is published in the chamber's name.
- Where the tool goes wrongA share of respondents is written as a share of members, so the base is wrong.
- The verification stepNext to every percentage write numerator and denominator; check the denominator matches the draft's word.
More: what was measured and the full check
What changed, as measured
On tasks the tool does well, consultants with the AI tool finished 12.2% more tasks and worked 25.1% faster, with better quality.
Who was measured: Consultants at a global consulting firm, randomised to work with or without an AI tool on realistic analysis and writing tasks.
The caveat: Consultants on tasks designed for the experiment, not chamber staff; on a task the tool does not do well, those with AI were 19% less likely to be correct.
The check, step by step
- For every percentage in the draft, find the row in the table and write its numerator and denominator next to it, for example "150 of 240".
- Compare the denominator with the word the draft uses: 240 allows only "businesses that responded" or "respondents"; "members" is allowed only when the denominator is 1,200.
- Look up every absolute number in the draft, such as "750 businesses", in the table. If it is not there, delete it, or write out the calculation and its base so the reader can see it.
- Add the number of responses and the response rate to the bulletin next to the finding: "240 of 1,200 members responded (20 per cent)".
The course covers member services, surveys, research and documents that can be checked, with one accountable person making each decision.
We can help you with this process: “Business chamber AI: Member operations”, 8 hours.
A documented failure pattern
A share of respondents reported as a share of members
A paragraph for the members' bulletin that an AI tool drafts for the chamber of commerce's research office, from the results table of a member survey.
Source: Conference on Language Modeling, 2024, and 3 more.
Why it is easy to miss
When this paragraph is quoted in a letter to a ministry or a press statement, the chamber speaks for 750 businesses that nobody counted. 62.5 per cent is 150 of the 240 businesses that responded, not of the 1,200 members: of the members, all that is known is that 150, or 12.5 per cent, reported difficulty, and about the 960 who did not respond the survey says nothing. The tool attaches the percentage to the word "members", which in a bulletin reads as interchangeable with "respondents", and then multiplies 62.5 per cent by 1,200. The figure matches the table exactly and the multiplication is right, so both checks pass; the error is in the base.
Where the failure pattern is documented: Peer-reviewed study at COLM (October 2024) on numeric errors in AI summaries; studies by the BBC (February 2025) and by the EBU with the BBC (October 2025) on the accuracy of AI assistants; Microsoft documentation for Copilot in Forms (May 2025).Sources: Conference on Language Modeling, 2024; BBC, 2025; European Broadcasting Union and BBC, 2025; Microsoft Support, 2025.The case above is ours, written for work in Albania; the laws it names were checked against the official text.
What it looks like
The survey tableMembers invited: 1,200. Complete responses: 240. "Have you had difficulty finding qualified staff in the last 12 months?" Yes: 150 (62.5 per cent of responses).
The AI tool's draft for the bulletin62.5 per cent of the Chamber's members, some 750 businesses, have difficulty finding qualified staff.
Translated from the Albanian original.
Courses for this type of organisation
What you leave with
The consultation pack holds the question, the sample, the response count and the deadline, the groupings with counts and the verbatim quotes. With it, every claim in the summary can be traced back to the members' responses before the institutional sign-off, even by someone who did not do the work.
Chambers of commerce
The catalogue lists all 27 together, with filters for format and time.