Commercial software packages can be a viable alternative for producing parts of an information system without using your own resources. Whether to purchase a software package or build your own is often a difficult decision for companies to make. The purpose of this article is to provide evaluation guidelines when making this decision.  

OVERVIEW

Packages are developed by vendors to provide general solutions to typical application areas and the package is usually described as being a complete information system. In reality, they are only tested computer programs or procedures that are sometimes accompanied by user procedures for input preparation. Because of this, the overall system, sub-system and administrative procedures may have to be added to convert a package into an information system before installation can be accomplished.

As with any other information system, the first objective must be to clearly define the business problem and information requirements. This sounds rather obvious but it is too often accomplished in a superficial manner. This problem is alleviated through the proper execution of a Phase 1, "System Study & Evaluation." Assuming approval of the information needs and a precise project scope, various approaches to solving the problem are now considered. It is at this time that approaches such as the purchase of a commercial software package can be explored.

For additional information on Phase 1, consult:

• Phase 1 - System Study & Evaluation
• Activity F - Develop System Approach
• Evaluating a Packaged Solution

HOW TO EVALUATE A PACKAGE

The first step for Systems Engineering is to determine the availability of packaged products to solve the business problem. A list of packages may be created by reviewing the advertisements in various trade journals and by checking with publishers which provide lists of available products. Most of the listing services provide descriptions of each package that assist in generally determining if the package covers the required areas.

There is a variety of publications that routinely prints product descriptions of various software packages:

• AUERBACH PUBLICATIONS (Pennsauken, NJ)
• CDW (Vernon Hills, IL)
• CompUSA
• COMPUTERWORLD'S BUYER GUIDES (Framingham, MA)
• IBM Products & Services
• PROGRAMMER'S PARADISE (Shrewsbury, NJ)
• PROVANTAGE CORPORATION (North Canton, OH)
• SOFTMART (Exton, PA)

Computer hardware vendors are also an excellent source for information about software products.

When the prospective list is completed, each vendor is contacted for sales materials. It will be helpful to provide each vendor with the project scope and information requirements, then ask that they demonstrate how their package fits your specific needs. Systems Engineering should also provide the vendor with a list of other questions to be answered. These questions should be as specific as possible so they will be easily understood and properly answered by the vendors.

It is important to discuss the business problem to be solved and the information to be produced. Avoid telling the vendor how the product should be programmed, defining techniques to be used and other such technical requirements. There are many ways of implementing software and being too specific in these areas may cause you to miss an excellent solution for evaluation. Since the vendor is in this business and you are not, it is possible they may know something you do not. Some junior or inexperienced analysts sometimes write specifications only to showcase their knowledge of a subject which in fact may be very limited.

Some suggested questions are as follows:

1. What business problems does the product address?

2. What documentation is provided for the User (Administrative Procedures), Systems Engineering (general systems documentation), Software Engineering (program documentation) and DP Operations (specific to the organization's hardware/ software configuration)?

3. Will the documentation provided be customized to meet installation standards?

4. What normal hardware/software configuration is required?

5. What is the cost of the product?

6. Does the package provide sufficient Data Base information to interface with other applications? If not, can they be provided and at what cost (if any)?

7. What education is required and the costs associated with this?

8. What materials or forms are required and the costs associated with them?

9. What changes to the product are required to meet the Information Requirements furnished?

10. Does the vendor provide modifications to the product or at an additional charge? If so, what type of modifications are typical?

11. Is there any maintenance or other on-going charge for use of the product? If so, elaborate on each specific charge.

12. Are there any other costs involved with installing and operating the product? If so, elaborate on each specific charge.

Also, has the vendor supplied a copy of the contract and a list of references.

A matrix or chart should be prepared by Systems Engineering to graphically compare the various packages strengths and weaknesses. The product criteria, product implementation and vendor evaluation should be one part of the matrix/chart while the vendors are the other. Then a grading scale is developed. In doing so, consider the possibility that some items may be more important than others.

Product integrity points to consider are:

1. Flexibility of product in application.

2. Integration with other applications.

3. Utilizes (or is adaptable to) organization standards.

4. Warranty.

5. Features.

Product implementation points to consider are:

1. Easy of use.

2. Not be an administrative burden.

3. Have wide acceptance in the industry.

4. Have an effective implementation program.

5. Easy to install.

6. Have organized reference and instructional materials.

7. Provide for vendor consulting to assure the proper use of the product.

8. Easy to maintain after initial implementation.

9. Have a standard bill of materials and services.

10. Have a well-defined maintenance program.

11. Have an effective "hands-on" training program.

Vendor evaluation points to consider are:

1. Have a good corporate reputation for reliability.

2. Have a professional attitude and a high credibility rating in the industry.

3. Reflect financial stability.

4. Provide for a staff with in-depth knowledge and background.

5. Provide for professional corporate facilities.

6. Be able to provide for a demonstration of the product.

7. "Practice what it preaches" - the vendor uses the product themselves.

8. Have an intimate knowledge of their products.

9. Be totally committed to the support of the product.

10. Provide additional services to support the product.

11. Easily accessible and available to its clientele.

12. Maintain effective communications with users.

13. Have long range plans for the support and development of the product.

14. Have long range corporate objectives.

After the materials are received from the vendors, Systems Engineering should carefully review all items provided. Delete vendors from the prospective list if their package does not even come close to satisfying the requirements. Then contact some of the references for the remaining vendors. When calling a reference, ask specific questions. Have a list of questions prepared before the call and carefully record the answers either during the call or immediately after the call is completed.

Some suggested questions are as follows:

1. Does the product perform as stated?

2. Did the vendor provide file layouts or do they have some other means to conveniently interface with the product?

3. Was the documentation as complete and correct as stated?

4. Did the product require more, less or about the same amount of modification originally planned?

5. Does the product require an inordinate amount of maintenance?

6. When a problem arises, does the vendor service the problem?

7. If the vendor services the problem, is the response prompt?

8. Who executes the required modifications?

9. Were there any problems encountered during implementation? If so, what were the problems, how were they resolved and by whom?

10. Is the customer generally pleased with the product?

11. Was the cost of the product as stated or were there hidden costs (elaborate)?

12. Would the reference allow your organization to visit their site to see how the package is operating?

After the preliminary investigation, the findings are reviewed, the list of prospective packages for further study is prepared and the matrix/chart is updated. Each of the remaining vendors is contacted and a sales presentation is scheduled. Each vendor should be asked to supply their documentation for evaluation, either privately or in the presence of their representative. It is often best to review the documentation with a representative present since questions can be answered as they arise.

One very important and often overlooked point to consider is in the Data Base area. A clear definition of elements is essential, otherwise, reports produced from the package will be meaningless. Quite often, the vendor's description is different from that of the purchaser, e.g., "ORDER DATE" could mean the date an item was ordered by the customer, the date the order was received, or the date the order was processed. There are three choices at this point.

Since the package is used to implement the system, it should be documented like any other system, hence it is a consideration in the selection process. The System, Sub-Systems, Procedures, Operations, Inputs, Outputs, Files, Records and Data Elements are required to be documented using the organization's standards.

When a sales presentation is scheduled, Systems Engineering should assure that all appropriate decision making personnel will be present. This group would include selected Users, the the project team, Systems Resource Management, Data Resource Management, and DP Operations.

During the sales presentation, carefully review all of the material presented. Be prepared to resolve all possible questions during the presentation. After the formal presentation, carefully review the product documentation. Make use of the checklist to determine the completeness of documentation provided by the package and which components must be developed or supplements prepared in-house.

After the sales presentation process is completed, the findings are again reviewed and any unqualified packages are removed from the prospect list. Now Systems Engineering must evaluate which of the remaining packages will best fit the User's needs. It would be a good idea to have the users involved in this process and any presentations that are conducted.

When this process is completed, Systems Engineering and Project Management prepares a "make versus buy" comparative analysis. This becomes the basis for the purchasing decision.

When approval is received, the Purchase Agreement should be reviewed by the organization's legal counsel. Since this review process may take a reasonable length of time, agreements should be submitted for review as early as possible. Remember that the agreement must state specifically all of the items that have been agreed to by both parties. Most contracts state 'this is the only agreement between the two parties and all other agreements either verbal or written are not valid'. Obtain legal assistance with this part of the agreement preparation.

It is important that the agreement contain a complete "bill of materials and services" that accompanies the purchase of the product. This lists the specific materials that come with the product, e.g., manuals, forms, object code, source code, etc., along with the services provided, e.g., training and consulting. This list eliminates any confusion as to what exactly is to be delivered by the vendor and points out any hidden costs.

The package payment schedule should coincide with the delivery of the various parts of the package and the installation process. To assure the schedule is followed, the agreement should stipulate periodic review points where all concerned can discuss progress and assess project status. All of this may, on the surface, appear to be too much control, however, this is identical to the control expected for 'in-house' development.

As with any other newly implemented Information System, an audit should be executed to assure that the product performs as expected and the economic benefits are realized. The audit report should analyze the estimated development costs versus the actual costs.  

SUMMARY