What if much of the fragmentation discussed in our first report comes from a few hidden assumptions held by project leaders and funders in our field? Below we identify one such “risky assumption” that may impact several areas, as well as an idea for reframing.

Look for a revised report in the coming weeks. The ideas for these “assumption” posts come in large part from the feedback and ideas we have received in recent months from the community. Thank you all.

Risky Assumption #2: “When Funding is Scarce, Delay the ‘Research Design’ — and the Research.“ In times of funding scarcity (i.e., always), difficult decisions about priorities have to be made. Scarcity raises questions about what can be separated, and what can be sequenced.  While it may be appropriate to delay the execution of third-party research, we warn that it is dangerous to defer the “research design.”

Research design (aka the “blue print” of the study) can be just as important and difficult as game design. But don’t confuse the research with the research design. The research design is a planning phase, and is part of the design process — without data. We can think of the research design as a kind of “creative problem solving” that is required to convince ourselves — and others — that there was impact, what kind of impact, and based on what evidence and logic.

Difficult decisions about the sequence of design and research still need to be made, even assuming the research design is determined early.  One way to empower designers and producers is to make the strategy more visible, so that all stakeholders can understand how research is sequenced strategically. For example, consider these diverging viewpoints (we are not endorsing any of these as right, but do think all should be on the table):

1. Delay all research. Only fund research when the product shows promise.
2. Always allocate 5% to research. Such rigid formulas are not unusual for “program evaluation.”
3. Either 0% or 500%. The cost of some research designs go far beyond the development resources, leading some to take the attitude that anything less than full funding is a waste of resources.
4. Scaling is the only question worth investing in for research.
5. Quality is the only question worth investing in for research, since the market should handle everything else.

The greatest danger may come from repeatedly picking the same option without thought. To counterbalance, our field might push each game project to declare how they sequence and frame design and research, thus necessitating some (public) reflection about which combination is best for their situation. Similarly, funders with a wide portfolio of games should be pushed to reflect on how they approach research across a set of games; for example, some projects might be primarily about answering a research question, while others extend established research and so might need less resources to establish they are indeed aligning with a proven impact model.

…positive reframing: Always have a research design, but decide case-by-case on the investment to collect specific data.

Sound useful? Let us know what you think!
—–
Other assumption posts include: #1 (design as separate from research), #3 (the logic model is obvious – forthcoming), #4 (innovation is about game types – forthcoming), and #5 (there is one way to scale – forthcoming).

(This post was written by Benjamin Stokes and Gerad O’Shea.)