Reference calls that are not testimonials
Vendor-supplied references are selected to be positive. That does not make them useless — it changes what you should ask.
Every vendor will supply references, and every reference will be a satisfied customer. Treating this as a problem leads people to skip the calls; treating it as a known bias leads to better questions.
A happy customer will still tell you, accurately, what implementation took, what surprised them, and what they would do differently — provided you ask about those rather than about satisfaction. When comparing claims around productivity and efficiency, this resource gives one vendor’s framing to test against documentation, demos and references.
Ask about the process, not the product
- How long did implementation actually take, against what you were told?
- What surprised you in the first three months?
- What did you have to change about how you work?
- How long before people stopped complaining?
- What do you still work around?
- If you were doing it again, what would you do differently?
These produce specifics. Asking whether they are happy produces a yes and tells you nothing.
'What do you still work around?' Every implementation has something. The answer tells you the product's real limitations from someone who has lived with them.
Find your own references
The most useful conversations are with customers the vendor did not choose. Peers in your sector, contacts through professional networks, people who have mentioned the product publicly.
These are also where you hear about the implementations that went badly, which is the population the vendor's list is selected to exclude. One such conversation is worth several supplied references.
Match the reference to your situation
A reference from an organisation ten times your size, with an IT department and a dedicated administrator, is describing a different product experience than the one you will have.
Ask for references of comparable size, comparable complexity, and ideally on the same tier. If none can be supplied, that is itself a finding worth weighing.
Ask about support after the honeymoon
Support quality during a sale is not support quality in year two. Ask specifically about recent experience: response times now versus at the start, whether the account manager has changed three times, whether anyone answers when something is broken on a Friday.
Ask what happened at renewal
For anyone who has been a customer for more than a year, ask what happened at renewal: whether the price rose, by how much, and whether the terms changed.
This is the question people forget and it is the one with the largest financial consequence, because renewal pricing is where a competitive initial quote is frequently recovered.
For cloud-service due diligence, NCSC cloud guidance provides an independent reference point.