On Saturday, I spoke and participated in a panel at The Washington Post in D.C. to a conference of the Religion Newswriters Association. I was impressed with their organization. They clearly desire to understand the people they cover in their stories.

They graciously invited me to address the issue of religious research, addressing the question: What do we need to know when we evaluate religious research?

The panel included Roger Finke (from the ARDA and Penn State) and Dan Cox from Public Opinion Research. Each of us spoke for twenty minutes and then particiated in a Q&A session.

Here are the written notes I used when sharing with them.

Surveying Surveys: How to View Religious Research

Viewing religious research is actually no different than viewing or evaluating other kinds of research. The basics are the same. However religious research has the potential to stir greater passion and reaction more than some other kinds of research.

Also, religious research, like many moral issues, often deals with a halo effect-- people answer what they think they should say rather than what they actually do. Research, and religious research which I will focus upon, should be viewed (examined) from at least two levels-- the "big picture" view and the detailed view.

The "big picture" view revolves around three basic questions: Who? What? and How?

The Big Picture

First, WHO?

Two facets need to be examined.

First, anyone reading, reporting, using, or interpreting religious research should first determine who conducted the research. This is important as it provides a context for the development, implementation, and reporting of the research. Most researchers approach studies from certain positions and expectations (hypotheses). Religious research is not different in that most religious researchers approach a study from a faith or belief position. While they may attempt to conduct the study in an objective and unbiased manner, they may not be 100% successful.

Does partisanship show in the results? So, while everyone has some bias, this is okay if they (and you) are aware of it and the bias has not skewed the research study and it findings. (Example: Barna/Gallup/LifeWay all sell resources-- how does that relate to the information they are reporting?)

Wise users of religious research attempt to identify and evaluate who conducted the study and if the basic positions of the researchers/supporting organizations influenced the study (consciously or unconsciously). History will also provide some feeling as to the "trustworthiness" of the researcher/organization.

When I was in High School, I remember taking the ASVAP (Armed Services Vocational Aptitude Test). When I got my results back it said I would make a fine tank commander. I was pretty excited until I found out that two of my friends got back the same result--they would make good tank commanders. When we went to the recruiter we asked why, his response was simple--the Army needs tank commanders.

Well, if every survey points back to the resources published by the organization as the answer, that might be a cause for concern.

Transparency helps address this issue. For example, if the organization tells you the survey questions they asked and how they asked the question, you can see how the survey was structured. If they do not, that is a concern.

The second facet deals with who is reporting or releasing findings of the study. This often is different than the actual person/organization who conducted the study. It raises questions similar to those related to who actually conducted the research/study. Concerns about bias, a specific agenda, etc. are appropriate to examine and consider. The questions are often true and well done, but the reporting agency interprets the results. A reporter may interpret and report them differently.

Second, WHAT?

This question is also divided into two facets--sizzle and fact.

News stories tend to present the sizzle, topics and issues which are hot and are of current interest. Thus, certain findings may be selected from a larger number of findings. Often bad news catches the attention and travels fast. It may be selected from more positive news because it will attract attention, even though it is not the most important part of the research. For example, evangelicals seem to be in a season of self-loathing. We love bad news about what is happening because it gives us something to rally our cause around. Thus, sometimes the worst news is the most widely reported, not necessarily the most accurate.

One should be aware of this and seek to determine how the sizzle fits into the broader findings of the study. It is possible to select correct findings which may cause one to draw conclusions which would not be supported by the entire study.

This leads to the second facet: fact. This deals with what the actual or complete study says. One should determine what the entire study is saying, what the overall conclusions are. What are the major findings which are being presented? Are they logical? Are they what one would expect? How many questions were asked? You may be seeing the results for one question but the results for other questions might shed a brighter light on the findings. Now this does not mean that unexpected findings will not occur or that these should be ignored. It does mean that they should be examined to see if evaluation in the detailed view reveals any clues as to their accuracy.

An issue related to both sizzle and fact is: What is NOT being said/communicated. Has something been overlooked or ignored? Most research is done by people who mean well and work hard. Yet, even those who mean well can word questions in a way that gets to a conclusion you do not. The infomation is still valid-- "blank" percent of people answered "this" way. It just may not have answered the question you wanted answered. As such, not every story has to be a "gotcha" when you think the question is confusing. If concerns are identified, one should look for answers in the detailed view.

Third, HOW?

This question moves one into the detailed evaluation of the study. One should look critically at how the study was conducted. This may reveal weaknesses in the study and its findings or reinforce the strength of the study. To do this, one must look at each phase of the research process. Unfortunately, the research design and detailed information related to sampling, instrumentation, response/participation, and analysis are often not given adequate attention in reporting a study. You don't have to report them, but you should have confidence in them. Some information is proprietary, you have to decide if you have enough information to proceed. The wise user of religious research will attempt to evaluate all of the details for which they can obtain information.

So, let's look at the research process.

The Detailed View

The research process may be conceived as a triangle with each phase of the process building upon the foundational first phase-- clear identification of the purpose of the research study. (See Figure below.) Movement of the process is from the base to the peak of the triangle. The peak of the triangle is application of the findings. Each of the phases is important! One cannot say that anyone phase is more important than the others since failure or weakness at any phase results in a failed or flawed research effort. However, the first phase is foundational in that the information related to it directly impacts many of the other phases and is necessary for these phases to be completed. Let's look at what should be considered in relation to each of the different phases.

1. Primary Purpose of the Study

Was the purpose of the study clearly stated and was it focused so that the study did not stray? (This is often an internal step.) Was the study comprehensive, in that it covered all of the relevant issues related to the study's purpose? Now, many polls are not comprehensive and they do not try to be. They are simply a question asked of a certain sample at a certain time. It is OK that they are not comprehensive-- but don't treat them as such.

Were there limitations to the study which were clearly identified? Studies which lack focus and comprehensiveness often result in weak findings or findings which have "gaps." If a study clearly identifies that it is limited to certain groups, topics, etc., then it should not be judged negatively for this. However, the findings of such a study should be communicated within the context of these limitations.

2. Data Collection Strategy

This phase involves two elements:

(a) From whom is the information desired? (Define eligible respondents) and

(b) What methodology is appropriate for the type of information being sought and the persons from whom it is desired?

It is important to know who the respondent group is (participants are). Is the study population from whom participants will come the appropriate group (all Assemblies of God pastors)? This impacts the findings. It is also important to know how the sample of participants was selected. Was the sample a self-selecting one or a properly constructed probability sample? (Beliefnet online polls are fascinating, but not that different than an O'Reilly poll.) Large numbers of respondents do not necessarily equate to sound findings though they are often used to try to convince others of the soundness of the findings..

What methodology was used? Is it appropriate for the respondents one is desirous of including in the study? If ethical issues were identified, did the methodology adequately address them?

If the methodology and sample/respondent information is not reported, one should be wary of the conclusions and reporting. Even brief articles should give some indication of these important aspects of the study.

3. Data Collection Instruments

Examine the instruments used to collect the information. Were they valid? Were questions clear and unbiased? This is in the eye of the beholder-- sociologists and researchers will tell you with great confidence that they are right and others are wrong-- and you will believe him or her until you meet the next well trained and respected expert with an entirely different view.

Were there items that could be interpreted differently by various segments of respondents? In the religious world, different groups may interpret the same word/phrases differently. When there is more diversity in a sample, it often means that there is a greater likelihood that differing interpretations will emerge. The validity of the findings could be impacted. Did the items address all salient facets of the issue? There is nothing wrong with questioning the questions, but there are also many reasons to ask question in different ways. Sometimes the questions confuse the answer and hence the findings/conclusions.

4. Data Collection Implementation

Response bias. One should examine who the actual respondents were. What was the response rate? Did some segments of the original sample respond better than others, thus skewing the participants and findings? One could start with a very strong sample and, due to response bias, end up with a biased group of respondents (and findings).

Timing. When was the study actually conducted? The timing of a study can impact findings if certain major events happen right before or during the data collection period. For example, the measurement of church attendance right after 9/11 would be skewed. Many political surveys this season done pre-Sarah Palin would not be relevant now.

5. Data Analysis

Was the appropriate analysis of the data conducted? Was the data fully analyzed? Would further analysis have been appropriate or helpful?

6. Communication of Findings

Were the findings of the study communicated in a logical order with clarity and accuracy? Were charts and graphs constructed in a way to accurately and clearly communicate? (Labeling, true zero point, easy to understand, etc.) Did the charts/graphs overly emphasize a point/issue?

To reiterate something covered in the "What" section, one should consider if the findings make sense and are reasonable--given the respondents, the methodology, and the instrumentation.

Conclusions

So, correct use and reporting of religious research findings involves being critical (not in a negative sense) of various aspects of the research. One of the reasons for publishing research is to obtain critical evaluation of all aspects of the study. This is one way that the research community moves the research process forward. It will not be easy for a user to evaluate all of the issues identified, but examining as many as possible will enhance the soundness of the use of the research. It is okay, and really essential, for one to question all aspects of research.