METHODOLOGY NAVIGATION

This represents the concluding phase of the ISEM project. It is performed either after Phase 8, or if a project has been discontinued earlier on (such as after Phase 1, 2 or 3).

The formal deliverables resulting from Phase 9 will be the "System Evaluation Report" which is reviewed by Quality Assurance prior to being issued to management.  

GENERAL DISCUSSION

During Phases 1, 2 and 3, Systems Engineering prepared the "System Study and Evaluation Report," "System Design Manual" and "Sub-System Design Manuals." On the basis of these reports, management either approved or disapproved the project for design and implementation of the system.

If the project was disapproved, the reasons are outlined during this phase and placed with the original documentation for future reference. If the project was approved, this approval was given as a result of certain expectations outlined within the reports. The purpose of this evaluation is to determine if these expectations were realized.  

THE BENEFITS OF PHASE 9

"Unless we learn from history, we are doomed to repeat it." This may be an old adage, but it is certainly true. Many organizations tend to regard project reviews as a waste of precious time and resources. Consequently, systems are installed without any type of follow-up. Because of this, many of the benefits were never realized. For example, clerical savings may have been part of the justification for a new system, but operating management did not reduce or reallocate the staff to realize the benefits. Without follow-up, "Parkinson's Law" may occur where "work expands to fill the time available."

As another example, a new operating management, created as a result of organizational changes, may not be utilizing the information provided by the system. They may be unaware of, or lack knowledge about the system. In order for management to have confidence in their systems organization and the systems installed, they should have tangible evidence that their significant investments are producing the promised results.

The systems organization can realize many benefits from an "ISEM Evaluation." It is not just a platitudinous statement to say that "we learn from our mistakes." It is a clear and established fact. A detailed audit provides Systems Engineering and Software Engineering with an opportunity to review their estimating and design skills. This knowledge, along with historical data, can be gainfully applied to new systems and projects. As such, estimating guidelines can be updated.

A carefully executed audit can also add to the credibility of the systems organization by showing how well they performed and that they can account for their actions.

Finally, as a result of an audit, the systems organization may find that the new system is simply not functioning in spite of systems maintenance and revisions. In this event, the audit may indicate an entirely new approach should be taken as opposed to continually fighting the problem.  

"WHEN" SHOULD YOU CONDUCT PHASE 9?

The System Audit should be scheduled for execution after the system has operated for an adequate period of time. Typically, this will be between thirty (30) and ninety (90) days after Phase 8. In some instances, it may be necessary to conduct more than one audit, depending on the timing of implementation and execution of the various sub-systems within the system. For example, User Management may want a detailed accounting of project costs immediately after each Phase 8. The actual system evaluation can follow thereafter.  

"HOW" SHOULD YOU PERFORM PHASE 9?

Ideally, the audit should not be performed by the same individual(s) who developed the system. A neutral third party should be involved who can audit the system and project objectively.

The steps required to execute the phase are similar to those in Phase 1, Activities B and G.

• Phase 1, Activity B "Current System Analysis" - the intent of this activity is to study the existing system, sample work, and evaluate strengths and weaknesses. This same type of work is performed in Phase 9. Here, the Systems Engineer uses the systems documentation from the various phases to walk the engineer through the system.

• Phase 1, Activity G "Prepare System Evaluation" - the intent of this activity is to estimate and schedule the remaining work effort, and perform a Cost/Benefit Analysis. During Phase 9, the Project Manager examines actual time reported, costs incurred and delivery dates. A final Cost/Benefit Analysis is calculated based on actual data (not estimated).

The formal deliverable resulting from Phase 9 is the "System Evaluation Report" which details the findings of the System Audit. This is then reviewed with management.  

DESCRIPTION OF PHASE ACTIVITIES

Activity A - Perform System Audit

Systems Engineering reviews the original system design documentation to become familiar with how the system was intended to operate. Next, a survey of the system is conducted to identify how the system is actually operating. Any differences are identified and noted as part of the survey. If any problem areas are identified, Systems Engineering works with the Users to determine the problem and possible solutions. If necessary, a Work Request/ Objective is initiated to resolve the issue.

Activity B - Prepare System Evaluation

Project Management prepares the formal Phase 9 "System Evaluation Report" consisting of:

• Phase Cover Page

• System Evaluation - a textual analysis of the completed system (as prepared by Systems Engineering).

• Project Evaluation - a textual analysis of the completed project. This should include sections describing "Lessons Learned" and the final "Cost/Benefit Analysis."

• O-O-M Estimate/Actual - detailing all project estimates, schedules and expenditures.

• Phase Review Checklist

Quality Assurance checks this report to assure conformance with standards. User Management is then provided a copy of the report for review, along with Information/Systems/Data/Enterprise Resource Management.