Rigour or Rigidity? The role of CAQDAS in qualitative research.
Helen Marshall
School of Social Science and Planning
Royal Melbourne Institute of Technology
Introduction
This paper asks what rigour means in the context of qualitative research and whether using computer assisted qualitative data analysis systems (CAQDAS) enhances rigour or stifles creativity. To do this it explores two debates, a discussion in Sociological Research Online of the potential use of CAQDAS in ethnographic research and correspondence on a e-mail list of the uses of ‘autocoding’. The paper concludes with a brief case study of an ethnographic classic, asking if it would have been made more rigorous or less creative had the author used a computer package to aid analysis.
The concept of rigour in Qualitative Research
It is possible to make a crude distinction between those who believe that there is an underlying unity in qualitative research which means that we can set out rules for doing good research, and those who see qualitative research as a set of competing schools or paradigms. For the latter group, universal standards do not apply.
Jennifer Mason’s recent (1996) textbook for qualitative researchers implies a ‘universalist’ position. Rather than giving detailed recipes for techniques, she poses sets of the questions which must be answered and decisions which must be made at various stages of the research journey in order to produce credible work. Her questions push researchers to think carefully about alternative positions and to demonstrate to readers that they have at every stage debated their chosen position and can defend their own choice. This is a view which I describe as ‘good housekeeping’, and I admit now to the belief that it is something to which all qualitative researchers should aspire.
Mason implies that there are overarching rules for doing good qualitative research, even though qualitative researchers come from many different disciplines and epistemological traditions and use many different techniques. If the important questions are addressed, and readers of the study advised as to why decisions are made, it will be possible to tell good, qualitative studies from poor ones. It parallels Miles and Huberman’s view that::
…it is possible to develop practical standards – workable across different perspectives- for judging the goodness of conclusions. Even if we happen to be dubious about postpositivist canons, we are still accountable for the rationality and trustworthiness of our methods (1994 p5)
Guba and Lincoln’s discussion (1994) discussion of the criteria used for judging research quality by four ‘competing paradigms’ in qualitative research takes a different view. The ontological and epistemological differences between qualitative researchers mean that they cannot agree on how to evaluate research. Those located within the paradigms labelled ‘positivist’ (‘the received view’) and ‘postpositivist’(or ‘critical realist’) agree that good research is marked by
the conventional benchmarks of ‘rigour’: internal validity…external validity (generalizability), reliability (in the sense of stability) and objectivity (distanced and neutral observer) (p114)
Constructivist researchers, sharing an epistemology in which ‘findings are literally created’ in the course of research evaluate qualitative research in terms of somewhat parallel criteria. They are, firstly, ‘trustworthiness’. which is composed of credibility, transferability, dependability and confirmability, and, secondly. ‘authenticity’. (p114). The elements of ‘trustworthiness’ are equated exactly with the elements of ‘rigour’ required by positivists and postpositivists.
Critical theorists, however, assess the quality of inquiries in terms of the extent to which they take the historical situation into account, and their potential for helping to change the world.
I do not intend to discuss the merits of Denzin and Lincoln’s characterisations of paradigms and standards. Instead, I point to their view that three of the four paradigms share some standards. I note too, that the standards they suggest tend to imply ‘good housekeeping’ – careful, rational decision-making which is recorded so that it is accessible by others, and scrupulous checking that the theoretical argument is supported rather than negated by the data.
There are still differences in the way qualitative researchers think about standards, but I argue that we can see a modest degree of convergence in the discussions of Mason and Denzin and Lincoln, and that the direction of the convergence is towards rigour as good housekeeping.
Kelle and Laurie’s (19950 comments encapsulate my view. They argue that although qualitative research starts from a different epistemological position from positivist quantitative research, it is important in both that we aim for rigour or validity through guarding as much as possible against self-delusion.
The aim of the validation process is not to prove the perfect agreement between results and ‘reality’ (an endeavour that would necessarily lead to an infinite regress) but to identify possible sources of error (Kelle and Laurie 1995 p22)
Whether I am correct or not in my view that there is a tendency to convergence towards rigour and the importance of checking for negative evidence, the discussion above shows that we have moved a very long way from fuzzy notions of undeniability and an ‘outline that holds’ (Smith 1978) as the standard for a good qualitative study. We have abandoned anxiety about the need for ‘scientific ‘analysis recorded in the preface of the first (1984) edition of Miles and Huberman’s sourcebook on qualitative data analysis, for a view which is both more grounded in technique (so that we now discuss techniques like autocoding and audit trails as well as issues of access and ethics) and more sophisticated.
The rapid replacement of the traditional means of record keeping and data management with the use of computers to store and organise data for analysis has contributed to this sophistication. In the days when analysis of large amounts of rich wordy data meant ‘shuffling cards or transcripts sheets into piles on the floor, reshuffling them, losing them, discovering that I was sitting on them and shuffling them again’(Marshall 1993 p152) one’s capacity for thorough searching for negating cases was limited by considerations of time and space. ‘Good housekeeping’ had a physical dimension. Today, computers can store enormous amounts of data, and CAQDAS enable infinite shuffling by a machine that is too stupid to lose material of its own accord. The next sections of the paper explore the meanings of ‘rigour’ for participants in two debates which belong in the 'analytical' phase of the research process outlined by Mason. Both raise the issue of how CAQDAS affect qualitative research.
CAQDAS Constrain Creativity? Coffey, Holbrook and Atkinson vs Lee and Fielding.
This is an example of debate at a fairly grand methodological level. It addresses the question of whether CAQDAS overall harm good qualitative research. Coffey et al in the online journal Sociological Research Online (1996) tend to the affirmative.
They argue that there is a tension between the pluralistic and polyvocal tendencies now found in ethnographic research and the standardising effect of CAQDAS. Postmodern influences have fragmented methodological approaches to ethnography and cultural research so that now ‘(o)ne can identify an almost carnivalesque variety of approaches’ which makes ethnography a problematic enterprise, containing ‘connotations of theoretical epistemological and ethical controversy’.(1.3). On the other hand, however, the growing use of computer packages designed to facilitate analysis of qualitative data contains dangerous potential to stifle ethnographic research by imposing rigidity.
Why do changes to technique pose the danger of rigidity? CAQDAS are dangerous for two reasons. First, their basis is the mechanisation of the traditional code and search technique. The packages will bring this underlying logic with them and universalise it as the logic in use (Kaplan 1964) for qualitative researchers. Secondly, CAQDAS impose the danger that all researchers will use grounded theory, which Coffey et.al. regard as inimical to genuine creative engagement with data:
…coding data for use with computer programs is not analysis …The growing respectability of qualitative methods, together with an adherence to canons of rigour associated primarily with other research traditions, can lead to the imposition of spurious standards…The categorisation of textual data and the use of computer software to search for them appear to render the general approach akin to standardised survey or experimental design procedures…no substitute for a genuinely ‘grounded’ engagement with the data throughout the whole of the research process (7.6)
Rigour, for these researchers, is a bogey when associated with cutting up data for close inspection. They seek the virtues of the postmodern turn in social science and relish the pleasures of polymorphous (even perverse) engagements with data in which the author’s theorising is open to challenge in a new way. They look forward to ethnographies in which the author uses hypertext to bring into contact with her data and her theorising a range of associated material which enables readers to deconstruct and reconstruct the text. Rigour here means making available all the ingredients in the cake and inviting others to make up their own recipe.
Lee and Fielding argue in their response (1996 ) that the view that CAQDAS compel rigour are completely wrong, that the presumed link between grounded theory and CAQDAS is grossly overdrawn, and that there is no logical connection between the potential offered by hypertext and the problems ascribed to CAQDAS, that the criticism that
First, the assumption that to code is to carry out a procedure more suited to surveys than qualitative research is incorrect. Coding, for Lee and Fielding is simply ‘data reduction’ and linking in hypertext can also be seen as data reduction. Both are simply strategies for automating part of data management.
I am uncertain about their latter point, but tend to agree that a code whether in an SPSS data matrix or the margin of an ethnographer’s field notes of itself is only a shorthand note to the researcher. What is important is what the researcher does with that note. Coding can be done in varied ways and for varied purposes; coarsely, in detail, descriptively, with a view to developing theory, or just to help keep track of which tape contains the whole conversation. There is no evidence to date that a common ‘code and retrieve’ approach to data managing will lead to uniformity in analytic approach. To argue that it will do so is to elide computer program elements with what people do in data analysis. Lee and Fielding and I agree with with Kelle (1997) who points out that code and retrieve methods represent ‘an "open technology" which can be creatively used in various contexts of hermeneutic work’(1.5). The marginal annotations found in many versions of the Bible allow those of many persuasions to create arguments. Kelle points out that we may regard these annotations as either codes (indexing) or hyperlinks (cross referencing) and that:
techniques of indexing or cross referencing are used simultaneously by all interpreters. Regardless whether they are more ‘orthodox’ and ‘dogmatic’ or more ‘liberal’, that means whether they take into account or not the polyvocality and diversity of biblical authors, their intentions and their diverse cultural backgrounds. And those (mostly historical) connections that can be found between data management techniques and hermeneutical schools which really exist point to the fact that ‘indexing’ (or ‘coding’) is extremely well suited to be used as a weapon against orthodoxy. (1997, 2.4)
Second, Lee and Fielding argue that the claim that grounded theory is becoming hegemonic is mere myth. Grounded theory like happiness means different things to different people. Even if it did not, there is evidence that researchers may use CAQDAS outside the grounded theory framework. A search using the Social Science citation index through studies where Seidel’s description of the avowedly grounded theory based program the ETHNOGRAPH was cited ( taken to indicate that ETHNOGRAPH was used in the study) found that only 30% also cited a work on grounded theory.
Lee nd Fielding are writing in a context where rigour is the avoidance of ad.hoc procedures for analysis through systematic data analysis (2.3) They clearly approve of mechanical means of data management, and have demonstrated elsewhere (1995) that researchers tend to use CAQDAS only as long as the mechanisms of the program fit with their epistemological requirements. In the debate in question, they note that hypertext ironically might prove particularly suitable for grounded theorists because grounded theory as a strategy for handling data:
Combines a tough-minded reluctance to collect more data than is theoretically necessary with an expansive concern to seek theoretically relevant data wherever it may be. The tools for doing this, memoing, theoretical saturation and theoretical sampling, depend on links, associations and trails which are difficult to maintain. Hypertext provides a technical means for doing so,. (4.1)
They are, however, cautious about the value of hypertext accounts of ethnographic research, arguing that, even with apparent polyvocality of the text with links to other texts, there is a single author who has chosen the items for expansion (eg back to full transcripts) and the database links. She sings all the parts, but multi-tracking makes her sound like a choir (4.4). There is also, they feel, a danger that the blurring of data analysis and interpretation may mean that readers get lost in hyperspace, going nowhere in a maze of links and connections.
There is some common ground in this debate. Both parties want researchers to maximise their readers’ chances of agreeing or disagreeing with interpretations. This comes close to my own view of rigour. Lee and Fielding’s version is modernist – there is some kind of ‘right’(‘better’ if not necessarily ‘true’) interpretation; good data management increases the chances of getting interpretation right and computers can help with data management. Coffey et al are postmodern, wishing to enable readers to comprehend both the complexity of the material which analysis has reduced and the contingency of the analysis, and convinced that hypertext will enable this. Their argument that existing CAQDAS will somehow rigidify and stultify analysis does not convince because they elide the programs with the users.
Autocoding – rigour or rigidity?
We turn now to the discussion of autocoding which took place on the QSR e-mail list earlier this year.
Auto coding is getting a computer package rather than a researcher to search for text and insert that text into some category. It is the quintessential ‘find and code’ operation which obviously in some ways mechanises qualitative data analysis. Thus it provides a good practical example in which to explore the contentions about rigidity in the earlier debate.
Earlier this year, an extensive discussion of the virtues and problems of autocoding went on in the QSR e-mail discussion list. Most discussants were expert users (especially of NUD.IST) and highly experienced qualitative researchers. What did they see as problems and virtues of autocoding?
To answer this question, I saved the relevant e-mails as they came in, and analysed them looking for themes broadly under the twin heading ‘virtues’ and ‘problems’. I used NUD.IST to manage the data but did not autocode.
Meanings of Autocoding
The eight participants in this discussion, originally had different views on what was meant by ‘autocoding’, and clearly used text search facilities in somewhat different ways. The range of meanings included:
As the discussion progressed, a distinction came to be made between ‘descriptive coding’ and ‘interpretive coding’ which generated theoretical insights. It was as a base for interpretative coding that autocoding was most controversial. It was seen as generally useful for the tedious descriptive business of ensuring that demographic details could be accessed readily.
Virtues and Problems of autocoding
It is clear from the discussion that these expert users of CAQDAS did not believe that autocoding could be used per se for coding for meaning, or ‘interpretive coding’. It could, however, be a useful precursor to the generation of meaning and theory.
There was, in the end considerable consensus on the issue. Autocoding was viewed as a tool, with virtues and problems.
I believe it can be useful in a "heuristic sense", but fewer problems? No. I would suggest different.
A nice debate about "autocoding" because it raises all the issues of what computers can and can't do. Perhaps it's because the term "autocoding" snuck in, but aren't we risking rejecting the use of mechanical processes as a first step to interpretive ones? I don't want to do that, indeed see search-coding as offering new qualitative tools.
Autocoding is just one of the tools to tease out meaning. Like archaeology - sometimes the camel-hair brush, sometimes the backhoe…Autocoding is a bit of a backhoe. It only gets used on some of the iterations.
A virtue of autocoding for these users are at the most practical level that it saves time. This is appreciated even by those who have reservations about other aspects of autocoding.
Qualitative researchers spend a lot of life doing descriptive coding. If I can autocode all the answers to each question in a structured interview and all the demographic details of participants, I will, thanks! Doing this mechanically is likely to avoid mistakes as well as save time.
Autocoding is very useful for fixed questions. However, it is no substitute for beginning the analysis by reading the transcripts again and again when the analytic approach is interpretative, and focused on meaning. But who has the time to do that?
It may also increase the reliability of analysis. One writer argues that it reduces the possibility of human error because computers (unlike researchers) never become bored and insert text into the wrong file by mistake. Another notes that computer coding based on searching for character strings allows replication.
The problem we "qualitators" have is that no two of us are likely to come up with the same set of codings. Autocoding (being mechanical) is at least reproducible.
The countervailing view is that autocoding (especially ‘interpretive’ coding) can get things wrong. This may be through mechanical error or misuse, and as the comments below suggest, it was misuse which concerned the participants most.
…We have visited this territory before on this list, of course, usually in the guise of discussing CAQDA programs as tools. Part of the problem is the term autocode, as though it is possible to equate an action that requires user input to make meaningful with a computer-based retrieval of a text string
…I support finding strings of text as a "heuristic" device that leads one into the data, that brings one into the data where it must then be interpreted, where the researcher must make a "coding decision". This can help resolve the problem of the false positive, but it leads wide open the problem of the false negative…The problem with "autocoding" is whether you stop with the "autocode", or you use it as a starting point. Among the many potential problems with the "autocode" is the extent to which it is equated with an "interpretive code".
Autocoding can be a useful tool, or it can be a debasement in the same sense that "shotgun correlations" and "shotgun causal modelling" are
…there still is something misleading about the term auto-coding. We frequently meet people who are left with the impression that qualitative data analysis packages offer artificial intelligence – that they interpret - that the software is a living breathing thing of its own. When we record our demographics, or organise placeholders for responses to structured questions - we're sorting key information - not interpreting, but setting up for a different direction at interpretation. And...text searching...is text searching...a powerful tool indeed, but as has already been established, by no means perfect.
Autocoding is generally agreed to be unsubtle. As noted earlier, it is a ‘backhoe’ and not a camelhair brush. As one writer noted:
It moves the focus away from the complexities and nuances of interpretative process toward a more superficial analysis that takes for granted an understanding of the general outline of people's stories etc. This is doubly a problem if the analysis is done by someone who did not do the interviews.
The same writer makes the point with an anecdote however, that manual coding will also produce unsubtle results if done only with a view to counting.
The issue of unsubtle coding is linked to the view that autocoding decontextualises material, thus possibly misleading the researcher. This was the most common criticism of autocoding.
Among the many potential problems with the "autocode" is the extent to which it is equated with an "interpretive code". You are correct when you argue that you cannot stop here, that you need "to go back through he text search and look at all the contexts in which the respondents use the word pain". I could not agree with you
more.
We have to be able to see data always in context...But what's the relevant context?
Using autocoding… encourages the researcher to think in chunks of decontextualised text. I find it hard enough to remember an interview as a whole as it is. But autocoding can be used to get things going so that it reduces the need to read through interviews as a whole. This makes it less likely that chunks will be seen in the context of what was said in the page before or after, or at the beginning or ending.
As one writer noted, there was considerable agreement on this. But there was a strong countervailing theme (raised also by those who had saw the problem of decontextualising) of autocoding as an aid to re-contextualising. The ‘backhoe’ might turn up unexpected treasure; it was not merely useful for mechanical descriptive coding.
I must say, though, that even though autocoding is a backhoe, I think it has always turned up a surprise or two. (Previous writer) points out the danger of decontextualized text, and it's true. But for me there is the other danger, of lost or buried text -- the phrase I don't notice because it is not where I expect it. The totally unimaginative autocoding device finds those unexpectedly-contexted phrases as well as the ones I would have predicted. Like the second-century Roman coins found in South India: by being discovered in a surprising context they establish an unexpected connection.
There are uses for this sort of first-stage scoop coding that are quite new to qualitative computing. I've found…that this sort of scooping-up of data gives a nice way of re-seeing themes - you get surprising juxtapositions of material, or passages you'd missed, and you can move between the finds and the context re-viewing, removing material that shouldn't be there and expanding context appropriately. And then by coding-on you can store these new ideas and pursue them. Wouldn't call that "autocoding", though...
I agree … that autocodes, particularly in large databases and from string searches, can give us a new perspective on the data base, and in my experience, they do uncover "lost" text -- material or points of view I have ignored. Precisely because the are so rigid, so mechanical, these forms of coding are likely to juxtapose incongruous chunks of text that I would not code because I have developed a "view" of the database as a whole.
The power of "auto-sorting" and text searching for me is the ability to approach the data from different directions -- to introduce challenges to our growing understandings of our data. (Previous writer) mentions that there are always surprises with text searches, I agree. There are also surprises in the text reading mentioned by (another previous writer). The ability to read data top-to-bottom document-by-document and then read across documents as you review responses to key questions AND to cut unpredictably into the data wherever we find a key word or phrase provides more opportunity for discovery.
This view contrasts with the pessimism of Coffey et al. For the e-mail list users, CAQDAS offer the potential for unexpected and creative insights rather than stifling rigidity.
A theme for two writers in the correspondence was that autocoding could be used to increase the rigour of analysis in a time-saving way, by allowing the checking of insights (no matter how generated).
One of the uses I most appreciate for autocoding is the quick-check-of-a-mid-project-idea - QCOAMPI? I recall that…, it occurred to me half-way through the coding that "door" was an important metaphor for my informant. I was able to autocode on the words "door" and "doors" and discover in about 20 seconds that yes indeed, this was an important idea. Because the software's autocoding facilitated the QCOAMPI, I could follow up on what might be hare-brained idea without sacrificing hours to it. And because I did, I confirmed something important. I was very grateful.
I support the point being made here as I did the same thing…It was very useful to run check searches to refine the variations that existed. And to do this with related terms.
In summary then, expert qualitative researchers feel that autocoding is a tool which can be misused, but equally can be of value. They certainly do not regard it as creating rigidity. The serendipitous effects of juxtapositions makes autocoding an ally of creativity while preserving rigour (thorough coverage and the possibility of checking) - if it is carefully driven. The misuse of autocoding is the assumption that the computer has ‘analysed’ the data, and that the collation of related materials is enough to deliver its ‘meaning’.
This misuse is what Coffey et.al fear will be the inevitable result of the growing use of CAQDAS. The examination of the views of experienced users shows that their fears are groundless. Computer programs may assist in making analysis rigorous if they are used appropriately – if the backhoe is used to do a sweep over miles of ground, and the patient slave takes the sieve and the camel hair brush to the dirt pile! To emphasise this banal point, I will end with an examination of a rather famous bad qualitative analysis, and ask if it might have been improved by the use of CAQDAS.
The Curious Case of Learning To Labour
Paul Willis’ Learning To Labour was one of the most influential studies in the Sociology of Education on the 1970s. It is an involving and passionately written ethnographic study of a small group of British working class boys in their final year at a comprehensive school which aims to show ‘how working class kinds get working class jobs’. It aims to demolish the simplistic ‘reproduction’ thesis, with its emphasis on social structure, and replace it with a ‘resistance theory’ that recognises and incorporates human agency.
It is also a classic example of unrigorous qualitative data analysis. The initial study looked at five distinct friendship groups of students,- three of which contained ‘nonconformists’ or rebels against the school culture , two of which were conformists or, in the boys’ own argot, ‘ear’oles’ As often happens, transcription money ran out (Willis 1975a). As also often happens, one group provided especially rich data. The twelve who identified themselves as ‘the lads’ became almost the sole source of the account.
We have some arguments in Learning to Labour that these twelve are typical resisters, but not much to convince us of this fact. And there is certainly nothing which justifies the extension from ‘the lads’ to ‘working class kids’, beyond a calculation that about a third of the total number of students in the schools were resisters. Neither is there much to justify the eventual thesis that ‘the lads’ are engaged in cultural production which simultaneously signifies rebellion and locks them into hard and heavy work.
Willis wants to convince his readers , in the absence of evidence, and even given evidence to the contrary…that it is neither school organisation nor working class culture that accounts for lads’ culture, but a sane adaptation to their anticipated fate in production(Reeders 1989 p 78)
With the hindsight that the collapse of resistance theory gives, many of the quotes used to illustrate the lads’ rebelliousness could equally be mined for comments suggesting the ambivalence which other researchers found amongst working class students. What appears to have occurred here is classic case either of ‘premature crystallisation’, or an ideologically blinkered argument. Willis certainly appears not to have not obeyed the advice of Miles and Huberman to
Trust your plausibility intuitions but don’t fall in love with them. Subject the preliminary conclusions to other tactics of conclusion drawing and verification (1992 p247).
Interestingly, Willis’ reflections on ethnography (1980) and participant observation (1978) both include the point that although the qualitative approach cannot be equated with positivist objective methods, it shares a need for rigour, for ‘the elimination of distortion, the cross checking of evidence and so on’(1980 p92). He argues that participant observation requires three linked strategies
(i)That the scope of distortion inherent in the participant observation problematic should be minimised as far as is reasonably possible by ‘holding to the natural’.
(ii)That a range of methods should be used to ‘cross-grid’ the evidence and so further reduce distortion
(iii)That finally the specific and finally irreducible problematic of the Participant Observation method should be used as a resource in a ‘self reflexive’ analysis…
(Willis 1978 p 194)
Ethnography, he argues has a real subject of inquiry, although we can know it only through our own ideas. This places demand on the researcher:
If your purpose is a fuller understanding and knowledge of this subject, then we must have some concern for the reliability of the data we use…It is perfectly justifiable to use rigorous techniques to gain the fullest knowledge (1980 p91)
We should assume then that the picture of the lads’ culture has been thoroughly checked. It is certainly true that, although Joey the leader possibly gets a little more airtime than the rest, the quotes in Learning to Labour come from a range of participants and are spread across group interviews, individual interviews, school and family settings. To that extent Willis has been rigorous. But his desire to rescue working class kids from the stigma of being victims of structure leads him, despite his own methodological strictures, to depict them crudely as either earoles or heroic resisters, ignoring subtleties, and to generalise wildly beyond the scope of his own data to
Would his work have been improved had he used a computer package for data analysis? My view is that Atlas-Ti, Ethnograph or NUD.IST would not have saved him from deciding that his twelve boys represented the entire working class and that the jobs they moved into were typical of all working class. The basis of that error lay in the earlier decision about sample type and size. The initial design was clearly intended to be comparative, but one group became the main focus. This, plus transcription problems (described in detail in the 1975 report) led to the analytic choice of elements from the overabundance of data as a focus It is possible that a computer program where reports on the percentage of coded data represented by a given theme were available might have given Willis cause to ask if he was overstating his case, but unlikely if the codes came from only one set of transcripts!.
Would computers have made it possible for him to create a more nuanced picture of the boys’ relations to school? Had the money been available for transcripts, computers might have saved time in data management and perhaps thus created a more complex picture of resistance. Even if the transcription budget gave out, the relative ease of making notes on the content of tapes using a word processor, then coding those notes might have enabled consideration of a wider range of data. It is likely that he would still have been seduced by the brio of ‘the lads’ and have made them the centre of his analysis early on. But if his coding has been assisted by a computer package, it might have been more thorough.
Could autocoding have made a difference? . It might have tempted him to seek comparisons more often through the data from the ‘ear’oles’ Had a computer search for a word like "laff" –‘having a laff’ was one of the main ways in which the lads asserted their cultural values against those of the school- created unexpected juxtapositions of text, would it have shaken Willis’ faith in the ‘lads=resisiters, rest = compliers’ equation?
I have no way of knowing, but I suspect not. Willis is clear that a broad theoretical orientation underlies every piece of research, even though researchers should attempt to hold their theoretical fire and ground concepts in data. He appears to have confounded his starting point with what he wished to emerge from his data – resistant working class cultural production. It is thus unlikely that he would have found more instances of compliance and less resistance. He was not looking for them. Resistance through cultural production was already his interest. His PhD thesis examined bike boys and hippies, finding both subcultures to contain elements resistant to capitalism (Willis 1978). Overwhelmed by data from the next study, he started with the resisters, and found only resistance. There were other studies preceding his (notably Hargreaves 1967, which Willis cited) which offered rather more complex typologies of orientations to school, but they did not lead Willis to question his own data.
Using CAQDAS might, however, have affected Learning to Labour in two beneficial ways. First, the chance to include memos with his coding might have enabled Willis at least to incorporate the fieldnotes on the other groups, and to reflect on how his love affair with ‘the lads’ might be shaping his theory. Second, if the data were being managed with the aid of a CAQDAS someone might have asked him about his categories. How many did he generate? How did he ensure that his coding was reliable? (did he code, or did he treat the data in some other way?) Was there an audit trail? Matters of this nature are now part of the ordinary polite collegial discussion of many researchers, in addition to the question ‘what themes are you finding?’ If this questionning had happened in the course of Learning to Labour, I think the end product would have been richer, more nuanced, as compelling a read, but in the end a more valuable product.
Conclusion
This paper has shown that for many qualitative researchers the heart of rigour is careful housekeeping and a clear-sighted willingness to answer the difficult questions – in other words to ‘name one’s shot’and be prepared to criticise it oneself and have it criticised by others. Those who feel that rigour is inimical to creativity, and that computing leads to the onset of rigor mortis, killing creative analysis, have be shown, both by argument and with evidence of ‘logics in use’(Kaplan 1964) grossly to overstate the case. CAQDAS can improve the rigour of qualitative research because they are tools which help extend the techniques of good housekeeping . Equally, they can help in the creation of junk research. The example of Learning to Labour shows that, just as anthropological folk wisdom says that researchers discover exotic cultures which mirror their own dispositions, so qualitative researchers will, in general, produce research reflecting their own (in this case theoretical) predispositions. The advent of CAQDAS has made it possible to ask ‘difficult questions’of ourselves in new ways, but does not impel us to ask them. Whether we are using cards or bytes, as Richards and Richards (1995) warn, the GIGO axiom always applies.
References
Coffey, A, B Holbrook and P Atkinson (1996) ‘Qualitative Data Analysis: Technologies and Representations’ Sociological Research Online vol 1 no. 1
http:<//www.socresonline.org.uk/socresonline/1/1/4.html>
Dey I (1995) ‘Reducing Fragmentation in Qualitative Research’ in Kelle op cit pp69-79
Glaser, B (1992) Emergence vs Forcing: Basics of grounded theory analysis Sociology Press Mill Valley
Glaser B and Strauss A (1967) The discovery of Grounded Theory: Strategies For Qualitative Research Aldine Chicago
Grbich C (1998) 'Are Computing Packages appropriate Tools for Generating theory from Qualitative Data?' The Australian Journal of Social Research Vol 5 No 1 July pp55-61
Guba, E and Lincoln, Y (1994) ‘Competing Paradigms in Qualitative research’in Denzin, N and Lincoln, Y (eds) Handbook of Qualitative Research Sage Thousand Oaks pp105-117
Kaplan, A (1964) The Conduct of Inquiry: Methodology for Behavioural Science Chandler San Francisco
Kelle, U (Ed) (1995) Computer-Aided Qualitative Data Analysis: Theory Methods and Practice Sage, Thousand Oaks
_______ (1997) ‘Theory Building in Qualitaitve research and computer Programs for the Managemtn of textual Data’ Sociological Research Online vol 2 no. 2
http:<//www.socresonline.org.uk/socresonline/2/2/1.html>
Kelle, U and Laurie, H (1995) ‘Computer Use in Qualitative Research and Issues of Validity’ in Kelle op. Cit pp19-28
Lee, R and N. Fielding (1995) ‘User’s Experiences of Qualitative Data analysis software’ in Kelle op. Cit pp29-40
___________________ (1996) ‘Qualitative Data analysis: Representations of a Technology: A comment on Coffey, Holbrook and Atkinson’ Sociological Research online vol 1 no 4 http:<// www.socresonline.org.uk/socresonline/1/4/lf.html>
Marshall, H Not Having Children Oxford University Press 1993
Mason J (1996) Qualitative Researching Sage London
Miles, M and Huberman A M (1984) Qualitative Data Analysis: A Sourcebook of New Methods Sage Newbury Park
________________________ (1992) Qualitative Data Analysis: an expanded Sourcebook 2nd Edition Sage Thousand Oaks
Reeders, E Orientations To School and Work (1989) Unpublished PhD thesis, Monash University Dept of Sociology
Richards T and Richards L (1995) 'Using Hierarchical Categories in Qualitative Data Analysis' in Kelle op cit pp80-95
Smith, L (1978) ‘An Evolving Logic of Participant Observation,Educational ethnography or Case Studies’ in Shulman. L (Ed) Review of Research in Education Vol 6 1976 Peacock Itasca
Strauss, A and Corbin, J (1990) Basics Of Qualitative Research: Grounded Theory procedures and Techniques Sage Newbury Park
Willis P (1975) The Main Reality Birmingham CCCS Stencilled Occasional Paper no 38
_______(1978) Profane Culture Routledge Kegan Paul London
_______(1980) Notes on Method in Hall, S; D. Hobson, A. Lowe and P. Willis (eds) Culture, Media Language Hutchinson/Birmingham CCCS London
_______(1977) Learning To Labour: How Working Class Kids Get Working Class Jobs Gower Aldershot
_______(1990) Common Culture: Symbolic work at play in the everyday cultures of the young Open University Press, Milton Keynes