Proof of Concept

Details
    Published on Thursday, 05 January 2012 16:29
    Written by Craig Sutherland



At some point you will want to start the process of evaluating NMS suppliers and equipment. The proof of concept (PoC) is an important phase of the NMS evolution process. Sometimes it is difficult to be impartial during this process. Organisations often have a long standing relationship with one of the vendors involved in the PoC process, and they may have intimate knowledge of your network and its challenges.



To get the most from a PoC, you need to furnish each perspective vendor with as much information about your network and application as possible. In many cases, you may even find that a vendor withdraws from the evaluation process because its solution does not meet the PoC criteria. The following template can be used to send to perspective equipment vendors before the PoC starts. You will need to adjust the PoC criteria section to suit your own need. 



However, I have included some relevant sections which are generic and should help to get you started. Another suggestion with the PoC criteria is to invite each vendor to write the PoC criteria based on the information you provide in the template document. This way, you can compile the best questions from each vendor into a single criteria document, which is used as a common template document for all vendors.



Download pdf of the template here Proof of Concept template

Sample PoC Template Starts here.



    Background Information and Company Profile

XYZ corporation manufactures/provides [description of what the organisation does]. Each perspective vendor is invited by means of a PoC to demonstrate their proposed solution fills the business and technology requirements required to deliver a comprehensive network [performance] management solution.



The organisation operates in a [hub-and-spoke/distributed architecture] with a data distribution centre[s] located at [name of site]. Redundancy is achieved by [Hot Standby Core switching architecture/multiple redundant WAN links/Fully meshed MPLS topology]. The core switching infrastructure is running at version [enter version release] and the routing infrastructure is running at [enter version release].



CPU and memory utilisation on the network devices is generally[Low/High/Critical]. The organisation operates [Monday to Friday/24hrs] and network management operations are [continuously staffed by the operations centre/are required continuously during hours of operation and support staff are on standby outside of these hours].



    Duration of the PoC

The XYZ PoC evaluation process will last for [1 week/3 weeks]. Central management equipment will be hosted at [name of site] which has full management access to the network. To minimise disruption, vendors are requested to install software off site as much as possible. Vendors are asked to complete the following form:



 Duration

Please specify the amount of time required to set up the equipment and configure the system to achieve the requirements outlined in section 4 (Evaluation Criteria)? ___________



Space

Please specify the rack space required to house the equipment in RU units. In addition, please specify the depth of the equipment and power requirements. _______________



Network Map

The vendor is requested to submit a network map [included in this document] showing the placement of its equipment and the interaction if any with other components on the network.



Addressing

XYZ corporation operates a [IPV4/IPV6] addressing scheme and has allocated the following subnet for use during the PoC evaluation. [Subnet Range included here with network mask and gateway]



Connectivity

The equipment will connect to [Our live Network/Our test network] and as such, the vendor is required to submit the ports and port ranges and protocol types required for connecting to the network. Where not well known protocols are used, the vendors is required to provide an explanation of the ports' requirement and purpose it serves.

Ports and Protocols required for PoC evaluation: ________________________



Access

The vendor should use the following user name/password combination for accessing the system if multiple user name password combinations are used.

User name: [User name_Here]

Password: [Password_Here]



Configuration Change

If integrating the solution into the XYZ network requires modification to the existing network's routers or switches. The vendor should detail the changes needed to the current configuration to accommodate the monitoring solution. XYZ will then assess the changes requested and may recommend changes which are in line with the organisation's [security/IT] policy.



Anti-Virus

The vendor is required to specify the primary operating system proposed for this evaluation. Servers configured with Microsoft Windows [must have anti-virus software provided by XYZ installed and run the current recommended service pack. It should also/should have up to date virus software installed and the latest service pack installed] before any connection to the XYZ network is possible. Vendors should provide details of operating system hardening (if any) which is provided by default.



PoC Setup Form [Ends Here]



    Purpose of the PoC

    XYZ Corporation is in the process of selecting a solution and a partner to provide a [performance/management solution] for [which part of the network does this cover/the whole network] the purpose of the PoC is as follows:

    Demonstrate how the proposed solution integrates into the existing network without disruption to day to day activities.

    Meets the objectives outlined in the evaluation criteria.

    Copes with the volume of traffic currently present on the network and has the ability to scale to meet the future demand.

    Intuitive interface, should be easily accessible to the XYZ staff.

    Understand limitations of the proposed solution within the scope of the PoC.

    Evaluate all features to understand cost plus benefit.

    

    Evaluation Criteria

 The evaluation criteria will be a four-stage process and is outlined as follows:

    Ease of set up and connection to the network. [Score Maximum 25 pts]

    Ease of use, logical work flow, context sensitive help [Score Maximum 25 pts]

    Ability to meets or exceed technical requirements [Score Maximum 25 pts]

    Additional features, cost plus benefit [Score Maximum 25 pts]

    

The vendor shall generate or capture screen shots for functionality assessment for each of the questions outlined in the evaluation criteria. This will be included as [hard copy/soft copy] at the PoC wrap-up meeting.



 [Evaluation Criteria Starts Here]

    Access

    The system was accessible from all areas of the network within the scope of the PoC and functionality was not limited by firewalls. Changing the user name and password for the default access account and adding additional accounts with less access worked. Concurrent user access worked, how many concurrent logins were demonstrated.

    

    Compatibility

    The proposed system should be compatible with the existing routing and switching infrastructure in the network. Any issues such as software versions and memory should be highlighted.

    

    Performance and Accuracy

The accuracy proposed in the solution should be stated. Please state factors which may affect the accuracy of the proposed solution where applicable. Any shortcomings introduced by the scope of the PoC should also included.  If any performance related issues were discovered during the PoC phase such as, additional load placed on routing and switching components, delay and packet loss as a result of the solution integration, this shall also be stated.



    Reporting

    A list of interesting reports generated should be included in the PoC. This will include the report name and include a sample where applicable. The system shall also state the limitation in terms of custom report generation based on user defined fields and which custom reports worked well in this environment. Reporting generation- what was the most effective means to generate reports, scheduled or Ad-hoc, manually or via third party solution.

    

  [Evaluation Criteria Ends Here]



    Sample Reports

Sample reports should be taken during the peak hours of traffic activity. It should clearly indicate the name of the report and the link or network segment the report was taken from and the time interval for the report. Reports should be made in portable document format (PDF) as a soft copy where possible.



    Revised Bill Of Materials

 If for some reason during the PoC evaluation it was determined that:

    The evaluation equipment did not have enough capacity/storage/processing power to meet the objectives in the PoC criteria, the vendor is invited to submit a revised bill of materials which covers the unforeseen demands of the network or an alternative solution.

    

    To meet the objectives of the PoC criteria, if additional equipment is required either from the vendor or a third party, then the revised bill of materials should include this additional inventory.

    

    Additional features are not realised in the original PoC criteria, then the vendor should include this in the revised bill of materials