Definitions

• Methodology: A general perspective toward research that includes what counts as research, what are appropriate methods for conducting research, what can be studied, how we come to know something, and what role values should play in research.
• Methods: The strategies or procedures used to answer research questions through collecting and analyzing data.
• Reliability: The consistency of your methods. In quantitative research, reliability is often described in terms of being able to reproduce the same results if the study was conducted again. If your methods are consistent, then reproducing the study should yield the same results. In qualitative research, the term reliability is either eschewed or used to refer to the degree to which researchers use methods and apply their theoretical frameworks consistently throughout the study.
• Validity: The believability of your research study. In quantitative research, results are valid to the degree with which they accurately depict the phenomenon being studied. In qualitative research, results are valid to the degree to which they resonate with your reader as being true.
• Threats to Validity: Perceived weaknesses in your methods that cause the reader to question your findings.
• Data Management Plan: A description of how you will incorporate your data into your analysis in a way that will keep you from drowning in your data. This includes how you will organize your data when you collect it so that you can find important segments without having to look through all of your data again; what data you will transcribe and what data you will merely take field notes on; what data you might start with in your analysis, what data you might consider second, an so forth; and how you will know when you have analyzed enough data and/or completed enough analysis.

What Goes In a Methods Chapter

The purpose of your methods section is to report how you collected and analyzed your data and to convince your reader that the way you collected and analyzed data allows you to answer your research questions and produce believable results. Methods sections are written in future tense when they are part of a proposal and past tense when they are part of a thesis or project report. Sometimes methods sections are separated into two major section: a section about data collection, and a section about data analysis. For quantitative research, the separation of data collection from analysis typically works well. However, in qualitative research, this separation can be problematic, because qualitative researchers often begin analyzing data as soon as the first data are collected, and based on the results of this initial analysis, make decisions about what data to collect next and how to collect it. If you interspersed data collection with analysis while conducting your study, you can sometimes still separate your description of data collection and data analysis by briefly noting which parts of your data collection are informed by analysis of previous data, and saving the description of how you analyzed the data at each stage for the data analysis section. If you feel that you cannot separate your description of data collection from analysis without misrepresenting your study, then you should probably describe and justify the steps you took in the order in which you did them. Even if you describe your methods in chronological order, though, you will still need to address many of the topics that follow.

Introduction

Your introduction should accomplish two purposes:

1. Describe the general approach you are using, and why this approach makes sense. For example, if you are planning to generate grounded theory, you would state that in your introduction and explain why you think grounded theory can answer your research question. The reason you need to tell your readers the general approach you are using at the beginning of the methods chapter is because once they know which approach you are using, they will be able to anticipate what issues you need to address in your methods chapter. This will help them make better sense of your methods chapter.
2. Describe the general layout of your chapter. Let the reader know what's coming.

Once you've done this, you're ready to describe the particulars of your study.

Data Collection

Depending on the context and purpose of your study, you may need to address the following topics related to data collection.

Setting: You will need to explain and justify the location of your study and the characteristics of that setting that are crucial to the study. For example, suppose I were studying eighth-grade students' understanding of ratios and proportions. I would probably want to describe the location of the school (e.g., inner city, rural, etc.), the type of school (e.g., public middle school, magnet junior high school), the size of the school, the student body (e.g., ethnicity, % of students who receive free and reduced lunch, % of students who speak a language other than English in the home), the mathematics curriculum that is being used, and so on. I would need to explain the process I used to select this particular school. I would also need to explain why this particular setting fits with the purpose of my research.

Participants: You will need to describe your selection criteria for choosing participants, the process you used to select participants, and important characteristics about the participants not included in your selection criteria. Using the example above, I would describe how I recruited 8th graders from Mr. So-and-so's class who are high achievers by individually contacting these students by phone and requesting that they volunteer for the study. After recruiting these students, I found that they are mostly female, white students, something I would need to include in this section. Then I would justify my selection criteria and process, and explain why they are appropriate for my research purpose.

Treatment or Intervention: In this section you would describe the types of experiences participants underwent to bring about change, if that is part of your study. You may want to include some of the actual materials in an appendix. Using the example above, I would describe the three 30-minute after-school classes that the 8th grade students participated in. I would describe the purposes of each class session, the types of activities used to achieve those purposes, and the reasons I chose those purposes and activities. I would include copies of the worksheets the students completed during instruction in an appendix at the end of my project or thesis, because these would take up too much space to include in the chapter itself.

Data types: In this section, you would describe the types of data you gathered, what instruments you used to gather them, and how they were gathered. Using the example above, I would describe the length and purpose of the pre- and post-interviews. I would report that I used a tape recorder to capture the conversations in these interviews, which I later transcribed. I would describe the types of questions I asked in these interviews. I would include a copy of the exact questions in an appendix. I would describe the location of cameras that I used to record the after-school classes, what the cameras focused on, where microphones were placed, and what portion of the data was transcribed. I would justify how these data are appropriate for my research purpose.

Data Analysis

While data collection sections are typically structured around the topics above, the analysis section can vary greatly from study to study. This is because the analysis section is structured to chronologically follow the steps you took to understand the data. Since no two studies are exactly alike, there is no common format to follow. Instead, you will create an analysis section that is unique to your study and that chronicles the steps you took during analysis.

When considering how big a step needs to be in order to warrant inclusion in the analysis section, it is better to err on the side of too much detail about your analysis than too little. While your readers don't want to know about every little detail, such as how you decided to code line 17 on page 4 of interview 3, they will want to hear about the steps that led to significant progress. For example, if you started your analysis by coding the pre- and post-interview transcripts for just one participant, and this led to a useful set of initial codes, you'll probably want to include this step in your description of your analysis. For each productive step you describe in your analysis section, consider explaining what you did, why you chose to do this step, what you learned from the step or what was produced by completing the step, and how it influenced your decision about what to do next. Then describe the next productive step, and so on. When you reach a step that you think your reader will find hard to understand, choose a short segment from your data and show how you performed that step on that particular segment of data. Your goal in this section is to convince your reader that every step of your analysis was reasonable and can be trusted.

What about the non-productive steps you took while analyzing your data? Most times your reader does not want to know about these missteps. Including every misstep can easily bog down your analysis section and make it difficult for the reader to follow the productive analysis you did. The one exception is when you make choices in how you analyze your data based on recommendations from the literature, and your analysis is not productive. Then you will want to record this misstep, explain what went wrong, and how you overcame the problem. The reason you include it is because it points to a possible methodological contribution made by your thesis.

Modifications for Proposals

If you are writing your proposal for a qualitative study, you will probably not be able to write a complete methods chapter, because, at the very least, you will be unable to predict the exact steps you will follow in your data analysis. Often decisions about analysis, and some times about data collection, can only be made after preliminary analysis. You have to wait to see what direction your data takes you. Your committee will understand this and will not expect your methods section to be complete. In most cases, you can provide your committee with a fairly complete data collection section. This will help them better understand what you are trying to do in your study and go a long ways toward convincing them that you are ready to conduct your study. In the analysis section, you can describe your data management plan. This will help them see that you have a plan for how to start your analysis and how to avoid being overwhelmed by your data. After presenting your data management plan, you should write a paragraph that describes what form you anticipate your results to have (e.g., a theory about what causes the phenomenon you are studying, a description of the common methods students use to solve a particular type of problem, a list of the ways that teachers cope with students failure, a set of recommendations for addressing a particular problem). You should also describe what contributions these results might make to the field. Help your committee envision what your completed study looks like. The more they are convinced that your study is doable and will lead to important findings, the more likely you are to pass your proposal defense.

Writing Tips

Reporting the "polished" truth. Graduate students often wonder just how honest/accurate they have to be about what they did and why. Sometimes graduate students make mistakes or stumble accidentally onto something important (experienced researchers do this, too!). How much of this do you have to report in your methods chapter? Do you have to tell everything exactly as it happened?

Nobody wants to be depicted as a bumbling researcher in their methods chapter. While it is not appropriate to misreport what you did to collect and analyze data, you are allowed to fudge some on why you did it. For example, suppose on a whim you chose to ask a particularly productive line of questioning in your interviews that yielded valuable insight into your research problem. Later, you find a paper (that you probably should have read before conducting the study) that suggests this line of questioning might be valuable. You don't have to tell people that you just got lucky; instead, you can cite this paper as the reason why you included that line of questioning. Similarly, if you fail to collect a particular type of data or ask a particular question that on hindsight seems negligent on your part, you don't have to include a statement saying that you should have known better if you can minimize the significance of the mistake by pointing to other researchers who have made the same methods choices. You can do this even if you never read or thought about these these researchers and their studies before you conducted your study. In summary, you are allowed to modify the reasons for doing or not doing something in your study to make the argument for your choice of methods sound more reasonable (and make you look better). You just aren't allowed to misreport what you actually did.

Justifying methods choices. Graduate students often feel the need to cite sources to justify their choice of methods. While this is a noble intention, you will probably not be able to justify all of your choices by referencing what others have said. True, you should try to find references for commonly used methods, such as case studies or semi-structured interviews. The library is full of quantitative and qualitative reference books you can refer to as you describe these methods. However, when justifying other choices, such as a the location for data collection or the number of participants in your study, you will most likely be unable to find citations to support your decisions. Instead, you will need to appeal to the reader's common sense about what would be appropriate for answering your research questions. This is particularly true in the analysis section as you describe the steps you took and why you took them. Also, sometimes you will have to justify choices based on the fact that you are only conducting a thesis research study (as opposed to, say, a multi-million dollar, NSF-funded research study). For example, you may justify interviewing four students because to interview more would produce more data than is appropriate for a thesis study.

Because it is often hard to determine exactly which choices you should try to find citations for, it is often best to not worry too much about inserting citations as you write your first draft of your methods. Trying to make decisions about whether or not to cite can bog down your writing and keep you from writing about choices for which no citations exist. Once you complete a solid draft, you can talk with your advisor about what parts need citations.

Handling threats to validity. No study is bullet proof. Every methods section has potential validity issues. The trick to writing a solid methods section is to anticipate what threats to validity your reader might see in your study design, and then either (1) describe the specific steps you took to minimize the threat, (2) try to convince your reader that whatever is causing the threat is unlikely to have happened, or (3) make an argument for why this threat, even if it occurs, will have little effect on the validity of your findings. The more you are able to anticipate and minimize threats to the validity of your study, while both designing your study and writing about that design, the more solid of a case you can build for the validity of your study.

Discussing codes. If you did a qualitative study, it is likely that you spent weeks of your life creating and modifying codes. You may wonder how much of that work you should share. For example, you may wonder if it is appropriate or necessary to share your initial set of codes, your final set of codes, or perhaps some examples of your codes.

While it is essential that you share and discuss your codes with your advisor as you meet for help with your ongoing analysis, you should be cautious about including detail about your codes in your description of your analysis. The validity of your coding scheme is derived from careful analysis of your data, weeks of thinking that your reader did not participate in. It is usually not possible to communicate that experience in enough detail for the reader to appreciate and understand your codes. Without this detail, your codes will be confusing and time consuming for your readers to think about, and probably won't lead to helpful understanding about your research. In fact, your sets of codes could easily be misinterpreted by your readers, which might cause them to question your findings, thus undermining the validity of your study. Presenting coding schemes also takes up a lot of space and requires time for you to write and your audience to read. So you should only include a coding scheme or examples of codes when the following criteria are met:

• A discussion of the codes is likely to convince rather than confuse your reader about the credibility of your study.
• The length of the discussion necessary to help your reader understand the meanings of the codes is relatively short.
• Your reader won't be able to understand your analysis process without some explicit references to actual codes.

Often when it comes to codes, an appendix can be the best solution. Keep the discussion of codes general in your analysis section, with perhaps an example of one or two codes to explain the steps in your analysis that would be hard for your reader to understand without illustrative examples. If your committee is not convinced by your discussion and believes a list of codes or an example of coded data is necessary, put that in an appendix. This makes your analysis much more straightforward and readable.

The one exception to the above discussion is when your final set of codes is actually your results. For example, if I were to conduct a study to identify the types of mathematical telling a teacher does while engaged in inquiry-based instruction, my final codes might actually be names for the types of mathematical telling I saw in the data. Thus, I would want to present this list of codes. However, this presentation would take place in the results chapter, not the methods chapter. So even in this case where the list of codes has to be reported, this list would not appear in my analysis section.

Common Graduate Student Mistakes

Forgetting to justify choices in methods. It is common for graduate students to get so involved in trying to clearly describe what they did that they forget to justify the methods choices they made. Make sure that you go back and reread your methods section with the particular focus of determining whether or not you have provided enough justification for why you did what you did.

Leaving out important information about your analysis. Graduate students often have to make large revisions to their data analysis sections, largely because they have left out important steps or justifications. One of the reasons for this particular mistake is that graduate students have only read data analysis sections from published research rather than from thesis studies. While published research can give you some idea of what belongs in a data analysis section, the data analysis section in theses are typically much more detailed than those in published research. Because you are a new researcher, you will have to go to greater lengths to persuade your reader that you analyzed your data appropriately. So plan on writing much more in your data analysis section than what is found in published research papers as you describe the productive steps you took in your data analysis.