The latest Adobe penetration statistics have just been released, showing the "worldwide ubiquity" of the Adobe Flash player. These statistics are used by Adobe to demonstrate that their plug-in dominates the browser market, with penetration rates in "mature markets" nudging past 99%. Or so they claim.

This has never tallied with the statistics for large, corporate websites that I have had access to. In their eagerness to claim dominance Adobe are being over-zealous and a close reading of their methodology can be quite revealing.

Where did you find these people?

The Flash penetration survey is conducted in eight countries quarterly, with a further five "emerging markets" surveyed bi-annually. The survey is based on a very small sample size given the overall population - 1,000 people are included from the U.S and 400 from each other territory, with these results being extrapolated to represent the entire internet. That said, the survey is fairly sound from a statistical point of view, as care has been taken to take a representative sample across genders and age groups.

The problem is whether or not these people are representative and how they have been recruited. This is where things get fishy. The survey uses Lightspeed Research and Global Market Insite (GMI) to collect the data from a pre-screened pool of registered subjects. These organisations incentivise people to register and take part in surveys - i.e. they get paid - so there is good reason to believe that they are not representative of the wider population.

The source of these panellists is also unclear as Adobe claim that they are recruited from "in-person interviews, web partners, as well as banner ads" without providing any further details. Relying on un-named "web partners" and unexplained "banner ads" as a source for panellists is particularly troubling.

What are we counting here, exactly?

The core problem with the survey is that it muddles up computers and people. The research surveys found that 99% of the participants could read the Flash animations on the machine that they happenned to be using at the time. This is not the same as saying 99% of devices have Flash installed on them. This kind of rhetorical "slight of hand" does undermine the credibility of the entire survey.

Ignoring the corporate lock-down

Adobe admit that "the majority of respondents answer from their home computer". This is pretty significant, as it does not take the experience of the work-place into account, where users tend to have much less control over what is installed on their computers. Corporate lock-downs of multi-media plug-ins are common place, as are restrictions on the sites that employees can visit. This bias towards the home computer user tends to skew the results further in Adobe's favour, ignoring the mass of corporate users who access the internet on machines with far older versions of Flash on a nine-to-five basis.

Mobile devices

The device landscape is changing and more people are using mobile devices to access the web than ever before - 8 million people in the UK used an mobile phone to access the internet in the first three months of this year according to OfCom. Adobe do not appear to include mobile in their calculations where Flash's dominance is not quite so complete - the iPhone alone might be enough to pull Adobe's penetration ratings down a point or two.

Always test YOUR user base first

As with many vendor-commissioned surveys, the results of the Flash penetration survey are based on a truth that is spun out to a rediculous degree, undermining the whole exercise. Flash may well be ubiqutous for PCs, but Adobe do over-egg the pudding in trying to prove their case. As Adobe themselves like to point out, the only way of being sure of your audience is to thoroughly test your own site's user base before making any judgements about Flash's reach.

Filed under Rants, Strategy and UI Development.