Archive for December 2009

How to evaluate open source solutions

December 20, 2009

One of the advantages of open source is the flexibility and low barrier to entry.  This also becomes a disadvantage as often there can be many similar solutions and it can become challenging for the user community to select one solution. 

This challenge can be overcome by having a methodology to evaluate and compare different solutions.  I will present to you a practical approach that I have developed and used.  If you choose to use it or adapt it, please let me know how it worked for you and how you adapted it to fit your needs.

There are a few published methods out there:

  1. Cap Gemini:  Open Source maturity model
  2. Qualification and Selection of Open source Software
  3. OpenBRR: Open Business Readiness Rating

After looking at these and others,  I have customized a model that has worked for me.   I call it a Practical Open Source Maturity Model.

Practical Open Source Maturity Model


  • Age/Maturity – Look at news reports and projects website for details on when the product was introduced, when it was available as a stable release, etc.
  • Momentum – Look at recent releases, number of articles in 3rd party news, number of community members, time line for releases and how well its been met in the past, etc.
  • Features – Yes, take a look at your technical requirements and how well the particular solution meets your needs.


  • Install – Read or speak with references on how their installs went. Check documentation for initial setup, config, 3rd party installation consulting services and backup/recovery procedures.
  • Usage – Research experiences with day-day operations. Check documentation availability for ongoing configuration, security patches, upgrade path, time line of until when support/development will continue for the particular product.
  • Support – How do you get support and assistance? Forums, paid subscription, 3rd party commercial support ?


  • Modularity – Is the architectural technical design modular and easily extend able? What examples/references are there for extensions and customizations?
  • Standardized – Does the solution use standard protocols that inter-operate with other solutions and all users systems ? Are the standards used public and have multiple participants or a standards body ?
  • Development – Is there a strong development community that is responsive to the users ? Is there an established process for development, q/a and release ?

Thats the Practical Open Source Maturity Model.  Use it wisely, document everything and it will save you a lot of time, frustrations and serious downtime! 

In the future I will post my evaluations of Firewalls/Security Gateways, Network Monitoring & Management software, etc.

Do you have an evaluation model that you have used? How does it compare with the Practical Open Source Maturity Model ? Let us know!