Unconvinced Pete: Science Isn’t Automatically News

Science is an interesting word. On the one hand it describes the process by which we come to have a true understanding of the world. On the other it can be slyly invoked almost like a magic word, miraculously transforming phoney claims into ones that seem plausible or mild claims into much stronger ones.

If those who commission studies are found to exaggerate or lie about their findings, they’d soon get into trouble so that doesn’t tend to happen. Neither is that necessary for them to get a good PR result; it works just as well to release a factually accurate press release and let the media say the magic words themselves. Recently, EA came out with just such a press release about a study they commissioned to see if their EA Sports Active product met ACSM guidelines. There was nothing particularly unusual about the study, it being typical of the kind done for the kind of product EA Sports Active is and I use it as an example only as it is the most recent one that came to hand. The press release, too, presented the results in a measured — if occasionally vague — manner that is difficult to object to.

Examining what the study actually established, we can see from the press release that when the sixteen physically active subjects used the product, it was found to meet the ACSM criteria for a good workout. It was not compared to any other methods of workout, game or otherwise. I contacted Dr. Porcari, who conducted the study, for a few more details and feel it is worth adding that the subjects were selected as physically active because the exercises themselves were quite difficult; they wanted to ensure the subjects could complete them. They were also given time to practise and ensure they were proficient in the exercises before the measurement-taking phase of the study commenced.

While interesting, none of the above is particularly alarming, even if it at first blush it seems a little like ‘cheating.’ All it serves to do is highlight the scope of the study and in particular what it does not assert. Firstly, the results do not claim anything about how efficacious the game is compared to other forms of exercise. Secondly and possibly most importantly, the study does not intend to say anything about how effectively the EA Sports Active performs as a fitness product ‘in the wild,’ outside of the more controlled environment.

On balance, it’s probably a good thing for EA to commission a study that certifies that the product is at least functional as a proper workout (when healthy people use it regularly and correctly enough) and from what I can see they have managed that. Then again, it seems like just about the minimum one would expect them to do. I can’t help but wonder why, if they are truly interested in measuring the performance of the product, they wouldn’t commission much wider scope trials comparing to other products for efficacy and testing its effectiveness in everyday home use. Note the difference between the two words “efficacy” and “effectiveness” as you read on and see that EA are careful to use the former. The latter would have to include a large sample of people in a range of fitness levels who haven’t had the opportunity to practice to the satisfaction of the experimenters, for example, in order to be an accurate reflection. But there’s no need for EA to put aside such resources when they know their press release will net them headlines such as:

ACSM rates EA Sports Active as an effective workout (Destructoid)
EA Sports Active is Legitimate Exercise, Says Study (1up)
Fitness Study Gives Thumbs Up to EA Sports Active (Wired)
Study Endorses EA Sports Active Effectiveness (play.tm)
Exer-gaming with EA Sports Active proves effective, according to scientific study (GamingBits)
EA Sports Active Actually Works, Says University Study (Gaming Shogun)

Nice publicity, of course, and it could even turn out to be all true, but that’s sort of beside the point. There’s just not enough space in a headline to say ‘Small EA Commissioned Study Suggests Physically Active Adults Within a Particular Age Band Who Use EA Sports Active as Recommended and After Some Practice Meet ACSM Criteria For Exercise.’ As much as I’d like all headlines to be like that, for some reason most media outlets think that something to the effect of ‘Science Proves EA Sports Active Fit-tastic‘ is a bit snappier. I’m sure EA agree.

So here’s my suggestion: don’t just take the bait. If there’s a study that looks like it might have just been comissioned primarily to generate exactly the sort of headline you were just thinking of writing, take a pause from the keyboard. At least read the wording of the press release more carefully. Better still, contact the PR team and ask for more details of the study. Decide for yourself whether the findings are actually significant or interesting enough to warrant giving the publisher that kind of publicity instead of just trusting that if ‘science says so,’ it must be news.

Personally I think that a fitness product being scientifically shown to function as a fitness product isn’t particularly newsworthy, but that’s me. Ask a few good questions and your conclusions will be your own; don’t and you’re just playing the part of a doting dance-partner being led around in a merry PR waltz.

I’d like to thank the EA PR team and Dr. Porcari for providing me with some useful fact-checking detail for this article.