Software Evaluation Guidelines

Impartial Evaluation showing blinfolded judge

Impartial Evaluation

As I prepare to write a new series of software reviews, including follow-up reviews for most if not all of the programs I reviewed formally for publications such as Cadence, 3D Design and other similar magazines, I would like to come up with some useful guidelines to follow when reviewing software in general, and Computer-Aided Design & Drafting (CADD) software in particular. This will include both personal guidelines, as well as guidelines that I believe should be followed by most reviewers. I also want to write them in advance, prior to performing the reviews, to hopefully prevent any personal bias to taint future results. This will be a reminder of the principles to follow.

The best news is my belief that reviews should focus primarily on the positive aspects of a program, what it can do and can do well, instead of what it can’t. The main reason is that people often base their buying decision on what a program can do for them, and one of the main purposes of a well written software review is to help the reader and potential buyer decide whether to invest their money or not. There is enough negativity in the world that I don’t see the sense in making another contribution. However, at least one admonishing comment should be made. After all, no program has been released that can solve all world problems, yet.


I believe a four star rating works well. It should also include the option of providing zero stars. The rating should take into account the program’s official description, particularly, what it promises to deliver. If it doesn’t deliver on its promises, it should be a candidate for zero stars. How well it delivers will determine the final tally.

Likewise, the rating should not happen in a vacuum. Programs should be rated on a relative curve, compared to their peers or with programs that offer overlapping, common functionality. In order to score the highest rating, 4 stars, a program should have State-of-the-Art features not quite available in competing products. In order to get 3 stars, a program must have impressive, awe deserving features that go well beyond what is “staple,” common or expected. It should elicit “Wows” and “Bravos”. It must be awe deserving. I know this is a subjective area where great demo artists can make or break a company. However, good reviewers must be able to see past possible smoke and mirrors. Some demo artists are so good they can make AutoCAD LT look like the best invention since sliced bread (…not saying LT isn’t worthy of admiration…)

To get a 2 star rating or better, programs should support all the common Graphical User Interface (GUI) features of the platform they are officially supported in. A two-star rating is an average rating, and most programs should fall in this category. Getting a three or four star rating should be an uncommon honor which should mean, “Don’t think too much about buying the product. It works, and it’s worth it.” A one- or two-star rating should mean, “You may want to think about it further before deciding to buy.” A zero star rating means, “avoid this program like a plague, or don’t buy it, unless you have a discretionary budget available for donations to keep struggling software development companies alive.”

To get a 1 star rating or better, programs should include a good help system and online help, tutorials and documentation. They should either be intuitive or provide a video or introductory tutorial warning the new user about potential idiosyncratic behavior to be aware of in advance. I often like to see how far I can go with a program “without having to crack a book open,” or cry for help and visit the online F1 help system. It is important for reviewers to do their best capturing and reporting on this experience, before being exposed to any kind of even informal training.

Similarly, innovation and originality should be encouraged and rewarded by carefully studying the time line of feature release dates, including which programs first brought a feature to market. Given enough time, all programs tend to merge and become indistinguishable as software developers get enough time to mimic the behavior and features of other programs. However, bringing features to the public light first should be praise worthy.

Minimum Information Required

I’ve usually covered this in a section titled “Vital Statistics.” Here I include information about the authors, their website, contact information, price, delivery methods, platforms supported and minimum hardware requirements. I usually award extra points for great multi-platform support.

Furthermore, a good review is a result of putting a program through the paces of a real-world project or problem. Ideally, it will be performed by an expert who is aware of the program’s real, not perceived limitations. Reading the entire documentation is an entry-level requirement for a good review. For an excellent review, including the advice and feedback of experts and real users is a must.

I often like to call and interview the developers and main key players of a software company prior to writing my final draft for review by senior editors. I like to ask them two questions: “Please tell me the strengths and weaknesses of your product. Please focus on the weaknesses. If I find a weakness on my own, they will count twice against you, so do yourself a favor and “come clean” and be honest on your own, without much coaxing required. Likewise, you better be pretty thorough. In fact, because I like to give my reviews a positive “spin,” if you want to be treated kindly, you better make sure you give me correct ammunition of a realistic weakness list.” The second question is about the future direction of the industry in general and the product/program in particular. How a developer answers these two questions is very revealing.

Finally, a good reviewer will be up front and disclose any potential conflicts of interest by making clarifications such as, “Developer X paid for my expenses to travel to Y city and study Z subject and meet with the CEO, etc.” Or, as I often do, disclose one’s current and past affiliations. In my case, for example, I think it is fair that people know I’m one of the founding directors and past presidents of AUGI, because of the product knowledge and fervor that you naturally develop when you spend time participating in CADD Communities. Be careful, it can even be a quasi or real religious experience, and I mean this in both a positive and negative sense, as in blind DOGMATISM which doesn’t help promote an attitude of impartiality.

How You Can Help

I mean this blog entry to be a spring board to creating a better and more refined set of Software Evaluation Guidelines. It is Work In Progress (WIP). Please post comments making recommendations on how to make them better. In particular, please let me know what software review features you have liked and appreciated.

Well written reviews serve to inform readers about new technology and options available, and help them make wise decisions while saving time and money. Thank you in advance for your help.

It will be interesting to see the suggestions I get from different vantage points such as from end-users, developers and publishers. As you can imagine, each group has their personal agendas. If you don’t feel comfortable having your comments published, please feel free to email them to me directly. Again, thank you in advance for your help and feedback.

Explore posts in the same categories: 3D BIM CAD, Community, Hardware Companies, Software Companies

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: