Reference Strategy
When we are leading software evaluation efforts we normally ask the software vendors for two customer lists from the software vendor: one of its large, major customers and one of its customers that are a similar size to you. You really need to think through your strategy with regard to reference checks. They can be a colossal waste of time or excellent feedback. The strength of the reference calls is going to come down to the questions that you ask. And you need to determine what it is you really want to know.
Who makes these calls? Is it the whole team, a handful of people, or just the project manager? There is no reason for a whole team of people to sit through a phone conference with the poor person who agreed to give a reference. Invariably, all discipline will break down and the reference giver will be barraged with questions from the third row. Go with a two- to three-person reference team: one to ask the questions and two to transcribe the answers and to listen for things not asked or answered.
What is the strategy for determining who to call? Ideally, it is not necessarily a company that made the vendor’s “A” list. All of us put our best customers on reference lists, the ones we feel certain will say good things about us. With a bit of research (go to the vendor’s web site) you can probably find some “off-list” customers. Having said all of this, be thankful for whoever you can get. Getting customers to take the time to give you the information you want is not an easy proposition. Take what you can get.
The results of these vendor reference calls should be summarized in a grid or a narrative. You do not have to wait until the product demonstrations to make these calls. Oftentimes after the last vendor demonstration you will want to go right into the final evaluation process. Making these calls—which often take a while to coordinate—can be a real momentum stopper. Ideally, you should make these calls prior to the vendor demonstrations. There is no reason to wait until after the vendor shows you his products. Don’t pass out the reference call information until you get to the final deliberations; it may prematurely bias an evaluator against a vendor that has not yet had a chance to give a demonstration.