by NUWPArc Team

Research Methods Practice

This exhibit (2021) traces the process of training two graduate student Research Assistants to learn qualitative analysis. Context for this exhibit is provided by Project Manager, Kyle Oddis. The exhibit features process notes from two RAs (Shannon Lally & Joseph Wheatley) that were stored as documentation during project development for use in exhibits.

Qualitative Coding

Methods practice overview

  • RAs worked with three "Batches" of 48 syllabi records from the 1990s and early 2000s.
  • Coding practice was conducted using 4 primary codes based on definitions of Values, Beliefs, Goals, and Practices articulated in NUWPArc's Conceptual Design Framework (CDF).
  • RAs were given practice sets of syllabi and reviewed coding procedures with Kyle Oddis & Neal Lerner.
  • RAs coded PDFs of syllabi using several experimental methods and research tools, including Dedoose, which is the tool ultimately used to visualize analysis results in this exhibit.

Coding Counts

Total codes observed in Practice Syllabi & Batches 1-3

Process Notes

Applying Codebook Definitions

View the process notes Research Assistants took during regular meetings to practice using and applying the codebook. Meetings were held with Kyle Oddis & Neal Lerner to discuss coding practices and procedures. Kyle and Neal also conducted qualitative coding procedures on the same sets of syllabi to discuss reliability and validity with RAs.

"Values" Keywords

coding observation: Values often appear as something immediately required or immediately resulting in consequence instead of reflecting a learning curve; i.e., one strike with plagiarism/improper citations becomes an automatic F in the class

frequently coded: Citing Sources; Annotation; Conventions; Standard; Attendance; Care; Serious/seriously; Tardy/late; Responsibility; Time; Revision; Cooperation/collaboration; Tutoring/disability services; Participation; Format; Anonymity/identity; Honesty/dishonesty; Integrity; “Goals” (coded when it said "goals" but the language expressed values, see Joe's notes); Holidays

"Beliefs" Keywords

coding observation: Beliefs are often steeped in specific discipline/sub-discipline/subfield training; often thought to facilitate success but are moreso encouraged than enforced, not often followed by punitive action (unlike Values)

frequently coded: "Writing is..."; words/phrases related to teaching writing (what teaching writing "is"); feelings; inclusion of a quote from a source text

"Goals" Keywords

coding observation: Goals often set a minimum requirement or articulate a deliverable to be met within some specific timeframe; classroom, field, or program-specific

frequently coded: Effective; Minimum grade for department/passing; Reflect; Hope; "You will understand"; Understanding; Confidence; Key; "I promise..."

"Practices" Keywords

coding observation: Practices designed to support Values incrementally over a specified period of time; translatable across fields

frequently coded: Typical; "Journal"; Reading; Writing; Workshops; Discussion; Essays; Email; Conferences; Structure

Shannon's Process Narrative

June 1, 2021

During my time working in the archive project, I had to develop a regiment for how to systematically and consistently code documents. First semester, I was provided with PDFs of the documents that were just scans, so I was unable to interact with the texts. What I ended up doing was using color coded sticky notes in Preview (Mac) to code instances of practices, values, goals, and beliefs. While this was a quick way to add a code, it meant that I had to go back and count the frequencies of all four types of codes in all of the documents manually. I then took those frequencies and compiled them in an Excel spreadsheet. Once this process was finished, however, it was discovered that other people could not see the color of the sticky notes, and so there was no way to tell what code had been applied to a certain section. Second semester, we switched to using Dedoose.

 

Dedoose is an effective tool, but unfortunately I still could not interact with the text itself. This means that the image has to be highlighted for the application of codes, not the actual words themselves. If a page is crooked, the coding process is tricky. Dedoose is also slightly slow because when you highlight a section, it can take up to 10 seconds for the section to actually highlight. Do not try to apply the code before the section appears as being highlighted, otherwise the code will not apply. When I could highlight a whole section, I did, but when sentences started or began mid-line, I only highlighted the beginning of the sentence or section.

Joe's Process Narrative

July 1, 2021

Currently, I am in the processing of developing a regiment for coding projects, while also taking into account previous regiments created by former Archive Assistant, Shannon. I was provided sample Syllabi that I could utilize to practice labeling the four following codes: Value, Practice, Goal, and Belief. I use Dedoose to highlight and add comments to the projects. When necessary I will also add memos to the projects if there are any questions or specific reasons as to why I coded a certain portion of text. Highlighting in Dedoose can be somewhat tricky; if you highlight one sentence and want to highlight the following sentence, the following sentence will combine with the previous. Also, with specific documents you can delete a specific code but you cannot remove the highlight.

Preliminary Results

Selected slides from Writing Analytics Presentation (2021)