Response to Kris on “How do we ensure product quality”

I got asked by our CEO “How can you benchmark product quality?” This is a pretty broad topic, but these are my central thoughts on how I’ve been able to establish and maintain quality in our products.

  • The way a consumer tells if a product is high quality is if it does what they want it to do without crashes or unexpected results
    • Only a user can tell you what result is unexpected
    • Therefore to measure the quality of the product, you need to use it the same way a user uses it
    • Startup time, snappy performance both get seen by the users as quality
    • “Work Product” is also seen as quality – if the product works as designed, but lets the user put dark blue text over a black background, the quality will be seen as “low quality”
    • The Setup is the “first impression” for the product as a whole – therefore it must be flawless and fast, and preferably value-add.
    • Documentation, Localization, box quality, icons, and the registration process all form a users impression of “quality”
    • User Experience Designers are critical, as they represent the user throughout the development process – and ensure the product does what the user expects.


  • Quality needs to be owned by the development team. The QA team can report on the quality of the product as they measure it, but they cannot change the quality.
    • The development team needs to feel that high quality is THE most important “feature”, rather than something that is done at a minimal level to get QA of their backs
    • Time needs to be specifically allocated for bug fixing throughout the entire development cycle, rather than just at the end of the cycle
    • Developers need to understand that most of our users are not “dumb”
    • Quality can’t be added at the end of the release
  • Focus on quality has to be at every milestone
    • “Fake” milestones can / should be added to give extra opportunities to focus on quality
    • Executive leadership has to be focused on quality at EVERY interaction – not on the cool features at the start of the release, and only on quality at the end. A really cool demo that crashes / fails 10% of the time should get equal bad press on the quality as it gets good press on the good features.
    • Metrics can be useful indicators of quality, if the teams using them are committed to delivering quality products
    • The minute people’s performance start to be measured by the metrics, the metrics become useless
    • To be useful, they provide an extra source of information to a team that is already committed to producing good results


  • The most useful indication of quality is project-based testing – using the products as our users do – from end to end – with a wide variety of PC & Peripheral hardware
    • Having the developers take part in this process helps to anchor that quality does matter
    • Your own experience highlights how much this can expose (Kris had just tried to use one of our products for a real-life task, and found an issue)
    • Don’t leave it up to the developers to “create” the scenarios, however – have the PMs, or UEDs, or QA leaders create the scenarios – but ensure they are real-world scenarios. (For video, for example – Don’t pull imagery off the web of a wedding – capture people walking into an office building (as entering a church), speaking at a meeting (to emulate the minister), leaving a meeting room (coming out the church), etc – then pull together a “wedding video” complete with DVD chapters, etc.

There is a ton more that can be said – but if you’re doing the above right, you’ve got a pretty solid foundation upon which to build.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: