Implementer Selection

Choosing a systems implementer is often not a straightforward decision. You are entrusting your organisation to another company with at best conflicting objectives and at worst a strategy that is the polar opposite of yours. Potentially your people will be impacted, your processes changed and your ability to keep doing business challenged.

What then, is the best way of choosing an implementer and minimising the risk? I have driven the selection process for a number of very large projects including, the City of Cape Town, Tshwane (previously Pretoria), and the Irish Revenue Service amongst others and find that the best way of evaluating several suppliers is to use a weighted decision tree. There are a number of reasons for using this approach:

1.       It ensures that all factors are taken into account

2.       It makes sure that the final decision is objective and not based on personal preference

3.       It is auditable and justifiable

4.       It gives you an answer that allows you compare price with capability for each provider

5.       Using an objective technique like this takes the politics and personalities out of the decision and focuses the selection team on solving the problem rather than making a deal.

This method lends itself to competitive bids where each supplier has to be evaluated on the same basis. A weighted decision tree has the following structure: 

• Division 1 
  • Sub Division 1 
    • Question 1
    • Question 2 
    • Question x  
  • Sub Division 2  
  • Sub Division X 
• Division 2
  • Sub Division
• Division X 

Each division is allocated a weight the scores for each question is multiplied up by the weights to arrive at a weighted score for each vendor. I have posted an example under my downloads section to make this clearer. 

The first level tiers are weighted according to their relative importance, these will typically include,

·         Capability: Do they have the skills to do the job? 20%

·         Financial viability: Are they going to stay in business for the life of the project? 15%

·         Proposed Technology: Are the technologies being proposed appropriate for the task? 20%

·         Price: What are they charging to do the work? Has the charging structure obfuscated the mix of staff? 15%

·         Vision: Is the approach being proposed a long term venture for them or just a short term money spinner? 15%

·         Service and Support: What sort of help are they offering after the job is finished, and how well does it meet your needs? 15%

The percentages are example relative importances for each section. Under each of these headings there will usually be further subdivisions, for example under capability there is likely to be:

·         Previous Company Projects

·         Project stream lead experience

·         Proposed team member experience

·         Familiarity with proposed technology

And so on.

To an extent the structure of the decision tree will have common elements for all implementer selections; however the minor headings and questions will vary between projects.

Under each heading there will be a number of questions. It is important to think carefully about the things you want to know about the supplier and to not give any latitude for weasel word or slippery answers, be specific in what you ask and determine what would constitute a good answer. Try to avoid questions that generate lots of verbiage as it is boring to read and difficult to score objectively.

For each question determine how you will score it before it is published. Generally it is easier if each question has a similar scoring approach, for instance, if we are looking at the technology component question:

“Does the proposed technology exchange data with Sun Accounts?” the scoring could be:

·         6: Yes out of the box

·         5: yes but it will take a little configuration, less than 1 hour

·         4: Yes but it will take major configuration, less than 4 hours

·         3: Yes but we will need to customise sections of the code, 2-5 days

·         2: Yes: but not until the next release

·         1: No, not ever

I have used this technique on several assignments and had the vendors actually score the questions themselves, which can significantly reduce the time it takes to execute the process.  

The final part to consider is how to run the project. This will vary depending upon the size and nature of your organisation; however a workable approach is this:  

Establish a team to do the selection. This needs to include representation from all the stakeholders, if this isn’t possible for some reason then the stakeholders need to empower the team to make the decision on their behalf. Consider using an external consultant or facilitator to execute the process and ensure that it is objective. If the project is significant to your organisation with lots of fingers in the pie you will save yourself a great deal of grief by having someone involved that isn’t part of your politics. 

Establish the questions and the scoring mechanism at the same time as you put together the statement of requirements document for the vendors.

Again depending on your organisation size and type, you may need to involve procurement and audit. If so, do it early in the process.

The evaluation team should establish the weightings to be used for the divisions and then publish them to the stakeholders before the vendor submissions are received. This stops anyone trying to manipulate the model to give a “better” answer or to favour a specific vendor.

Each team member should read every vendor submission and score every question. This is intended to give a balanced view across all stakeholder groups and ensure that all needs are taken into account.

Once everyone has scored the submissions a consensus workshop is help to bring the scores together. Typically scores are within a couple of points of each other, but when different stakeholder groups have differing requirements the scores can be miles apart. The purpose of this workshop is to resolve these instances and make sure that all assumptions and requirements are tested and addressed by the vendor. Being thorough early on costs a lot less than taking a vendor to court later.

Most evaluations attach a huge weighting to price. This invariably means that the vendors play games with structuring the package to make it appear cheap. A better approach is to give pricing a low albeit significant weight and evaluate the whole service independently of the cost. By using the weighted decision tree approach you can now analyse the results to understand the differences between the vendors from a strength and capability perspective and then evaluate the price difference between them.

This is a bit of a dry subject for a blog, but I hope you have gained a flavour of a best practice approach. If you would like to use this technique in your company feel free to give me a call on 01420 588172 or contact me here.