Craig A. Mertler,
Bowling Green State University

Background

The Internet has had a substantial impact on the field of survey research (Shannon, Johnson, Searcy, & Lott, 2001). This is evidenced by the growing number of electronically administered surveys over the past several years. Web surveys—which Dillman (1998) has referred to as one type of “self-administered” survey—are an extremely promising method of data collection (Schillewaert, Langerak, & Duhamel, 1998).

Advantages of Web surveys include a high rate of response, short time frame for the collection of responses, and time and cost savings. The Web certainly addresses the need for a less expensive and more expedient method of data collection (Solomon, 2001; Heflich & Rice, 1999; Schillewaert et al., 1998). Furthermore, several additional benefits of using the Web for data collection have been identified (Carbonaro & Bainbridge, 2000; Mertler, 2001, 2002a, 2002b; Schillewaert et al., 1998; Shannon et al., 2001). These benefits include a faster response, protection against the loss of data, easy transfer of data into a database for analysis, cost savings, convenience for the respondent, the possibility of wider geographic coverage, and a potentially better response rate— although this “advantage” is not uniformly agreed upon by the community of survey researchers (Matz, 1999).

However, it is important to note, as with any method of data collection, there also exist disadvantages. These include the potentially nonrandom nature of the sample, unavailability of population lists, computer access to the survey, and various technology-related issues. Additional limitations include the inability to clearly define the population, lack of technological familiarity on the part of respondents or their willingness to use a computer to complete the survey, the potential for being able to identify respondents, and browser incompatibility problems (Solomon, 2001; Carbonaro & Bainbridge, 2000; Schillewaert et al., 1998; Shannon et al., 2001).

One of the most substantial concerns about Web surveys is the potential nonrandom nature of the respondent group (Mertler, 2001, 2002a, 2002b; Solomon, 2001; Witte et al., 2000). However, the issue of nonrandomness is not unique to Web-based survey research and can be addressed through the maintenance of an accurate list of population members, when feasible. Survey research professionals have suggested that Web surveys be used primarily with specifically identifiable samples such as “in-house” employee groups (Shannon et al., 2001). Alternatively, Taylor (2000) has suggested that we remember that online data collection is not based on probability sampling, but rather on “volunteer” or “convenience” sampling.

Methodological Issues Related to Web-based Survey Methodology

Beyond the strengths and weaknesses of this methodology, Carbonaro and Bainbridge (2000) have outlined several other issues with which researchers should be concerned. First, access to the survey must be as simple as possible for all respondents. The more complex the process of completing the survey, the lower the resultant response rate will undoubtedly be. Second, the process must be designed such that respondents of the Web survey are able to complete it with the same relative ease as if they had received a traditional paper version. Third, some sort of security system is required, in order to maintain the integrity of the data, but also to put at ease the mind of the respondent. Finally, completion of the survey must require only minimal computer skills—including the ability to use an Internet browser, enter a specific URL, use a mouse, and type on a word processor. Matz (1999) and Shannon et al. (2001) also point out that a Web survey must somehow be publicized. Some mechanism must be used to direct potential respondents to the actual URL containing the survey, either by providing a link to the URL in an email message or by providing the URL in a cover letter.

Psychometric Qualities of Web-based Versus Traditionally Administered Surveys

Although more and more studies comparing the effectiveness of electronic and mail surveys are being conducted, there exists somewhat of a void in the research literature on this topic (Underwood, Kim, & Matier, 2000). Often, the results of Web-based surveys differ when compared to written questionnaires and telephone surveys (Taylor, 2000), although Saphore (1999) found that there were no differences in the pattern of responses between a Web survey and an identical pencil-paper form of the same survey. Furthermore, he concluded that there were no differences in the psychometric qualities of the two forms of the survey.

In another comparison-type study, Matz (1999) compared the responses received from a Web survey with those received from a traditional paper-and-pencil survey. She found no significant differences between the demographic characteristics—such as age and gender—of the respondents completing the Web survey and those completing a mailed, written survey. Additionally, she found no significant differences between the content or pattern of responses for either group. The only significant finding—albeit, a substantial one—was that the overall response rate for the paper survey (43%) was significantly higher than that for the Web survey group (33%). Following his review of the literature, Solomon (2001) found this to be a consistent finding for studies which compared Web and mail surveys. A final result discussed by Matz (1999) was that the paper instrument seemed to provide more flexibility to the respondents. They could freely make comments about items they did not understand or felt were ambiguous. In spite of these last two findings, she concluded that Web surveys seemed to be a reasonable alternative—or, at least, equally suspect—to mail surveys.

Another example of research comparing the psychometric characteristics of these two survey methodologies was conducted by Mertler and Earley (2003). The researchers concluded that Web-based and traditional survey methodologies result in similar sets of psychometric qualities based on comparable obtained values for internal consistency reliability. Although not identical, the patterns of responses—as identified by contributions to the overall scale—across the two modes of delivery were also fairly comparable.

In a final example, Idleman’s (2003) results contradicted those of Mertler and Earley (2003), with respect to the similarity of internal consistency values. Idleman (2003) observed that entire mail surveys, as well as subscales within those surveys, exhibited higher reliability coefficients that those resulting from a Web-based version of the same survey, although the coefficients were well within the acceptable range for both modes of delivery.

Patterns of Nonresponse in Web-based Surveys

A viable and important source of error in surveys is nonresponse (Montez, 2003). The reason for this is that the characteristics of nonrespondents may somehow differ from those of respondents, potentially limiting the external validity of the survey’s results (Montez, 2003).

Solomon (2001) notes that some potential respondents have difficulty with the technology and give up early in the process of completing a Web-based survey, or when encountering complex questions. Others may often be reluctant to give out personal information, such as an email address.

In her study of nonresponse, Montez (2003) received 55 (25%) follow-up responses from 218 original nonrespondents. In analyzing the reasons given for nonresponse, five categories emerged. The five categories were:

1. simple, but polite, refusal to participate;
2. not enough time to participate;
3. change in professional position (therefore, the topic of the survey was no longer appropriate);
4. desire to only respond to surveys that are prepared as part of the business of national organizations; and
5. belief that the survey was poorly designed or did not truly capture the  desired topic.

Summary

Because of the technological skills required to develop and implement Web-based surveys, the driving force behind their use has not been survey professionals, but rather technology specialists (or at least someone with a background in technology) (Dillman & Bowker, 2001; Shannon et al., 2001). In order to harness the potential for using the Internet for the collection of valid and reliable data, those most knowledgeable about survey research methodology—specifically with respect to causes and consequences of survey error (Dillman & Bowker, 2001)—must take an active role in its development as a viable methodology for data collection. Specifically, research is needed in order to compare responses from Web surveys and those conducted in more traditional manners (Dillman & Bowker, 2001).

Purpose of the Study

The primary purpose of this study was to add to the knowledge base regarding the use of Web-based surveys as a viable means of collecting data for educational research. Specifically, the researcher investigated and compared the relative effectiveness, psychometric qualities, and response patterns of two versions of the same survey: one delivered as a Web-based survey and the other as a paper survey. A secondary purpose was to assess the patterns of nonresponse, particularly with respect to the Web version of the survey.

The research questions addressed in the study were:

Research Question 1: What are the psychometric qualities and response patterns for survey data resulting from traditional and Web-based delivery modes? How do the two sets of results compare?

Research Question 2: What patterns of survey nonresponse result from the administration of the same survey?

Methods

Participants

During the fall of 2002, the researcher surveyed the population of interest (specifically inservice teachers) with respect to their assessment literacy. The group of inservice teachers consisted of 197 teachers, representing nearly every district in a three-county area surrounding the researcher’s institution. The schools were selected based on convenience due to their geographic location. All grade levels and content were represented in the final sample. The teachers were randomly split into two groups: one group received a paper-and-pencil version of the survey; the other group was directed to a Web-based version of the survey.

Instrumentation

Both groups of teachers were originally surveyed using an instrument titled the Classroom Assessment Literacy Inventory, or CALI, which was adapted from a similar instrument called the Teacher Assessment Literacy Questionnaire (Plake, 1993; Plake, Impara, & Fager, 1993). This inventory is based on the Standards for Teacher Competence in the Educational Assessment of Students (AFT, NCME, & NEA, 1990). The CALI consisted of the same 35 content-based items (five items per each of the seven Standards) with a limited amount of rewording (e.g., changing some names of fictitious teachers, changing word choice to improve clarity, etc.), as well as 7 demographic items. The complete instrument can be viewed at http://www.bgsu.edu/colleges/edhd/LPS/EDFI/faculty/mertler/CALI.html or http://PAREonline.net/htm/v8n22/cali.htm.

The Standards for Teacher Competence in the Educational Assessment of Students (AFT, NCME, & NEA, 1990) were a joint effort between the American Federation of Teachers, the National Council on Measurement in Education, and the National Education Association. This joint effort began in 1987 in order to “develop standards for teacher competence in student assessment out of concern that the potential educational benefits of student assessments be fully realized” (AFT, NCME, & NEA, 1990). They were originally developed in order to address the problem of inadequate assessment training for teachers (AFT, NCME, & NEA, 1990).

According to The Standards (AFT, NCME, & NEA, 1990), assessment is defined as “the process of obtaining information that is used to make educational decisions about students, to give feedback to the student about his of her progress, strengths, and weaknesses, to judge instructional effectiveness and curricular adequacy, and to inform policy.” The Standards, of which there are seven, provide criteria for teacher competence with respect to the various components of this definition of assessment. The Standards for Teacher Competence in the Educational Assessment of Students consists of the following seven principles:

1. Teachers should be skilled in choosing  assessment methods appropriate for instructional decisions.
2. Teachers should be skilled in developing  assessment methods appropriate for instructional decisions.
3. The teacher should be skilled in  administering, scoring and interpreting the results of both  externally produced and teacher-produced assessment methods.
4. Teachers should be skilled in using  assessment results when making decisions about individual students,  planning teaching, developing curriculum, and school improvement.
5. Teachers should be skilled in developing  valid pupil grading procedures that use pupil assessments.
6. Teachers should be skilled in communicating  assessment results to students, parents, other lay audiences, and other  educators.
7. Teachers should be skilled in recognizing  unethical, illegal, and otherwise inappropriate assessment methods and  uses of assessment information.

The Standards acknowledge and specify the importance of teacher education and professional development in the area of classroom assessment (Brookhart, 2001). All 7 standards apply to teachers' development and use of classroom assessments of instructional goals and objectives that form basis for classroom instruction. Standards 3, 4, 6, 7 also apply to large-scale assessment, including administering, interpreting, and communicating assessment results, using information for decision making, and recognizing unethical practices (Brookhart, 2001).

The original instrument has been shown to have reasonable reliability with inservice teachers, r_KR–20 = .54 (Plake, Impara, & Fager, 1993). Additionally, the original instrument was subjected to a thorough content validation, including reviews by members of the National Council on Measurement in Education and a pilot study with and feedback from practicing teachers and administrators.

Procedures

The paper-and-pencil group received the cover letter and survey directly through U.S. Mail and was supplied with a postage-paid return envelope. The Web-based group received an email message containing the cover “letter” and a link consisting of the URL to the survey. Two weeks after the initial mailing of the paper version and posting of the Web-based version, teachers were sent a reminder about completing the instrument.

Due to the low rate of response to the Web version of the survey, a follow-up “survey” was conducted via email for the group of nonrespondents. This email survey consisted of the following:

 Analyses

Statistical analyses included the computation of internal consistency (i.e., coefficient alpha) reliability coefficients for the overall instrument , as well as independent-sample t-tests in order to compare the resulting scores for the Web and paper versions on each of the seven subscales (i.e., standards) and for the total scores.. All statistical analyses were conducted using SPSS (v. 11). Content analyses were also conducted for the teacher-supplied reasons for nonresponse in an attempt to classify them into thematic categories.

Results

The results that follow are presented by each individual research question.

Research Question 1:  What are the psychometric qualities and response patterns for survey data resulting from traditional and Web-based delivery modes? How do the two sets of results compare?

The return rate for the total survey was equal to 17%; the return rate for the paper-and-pencil administration was equal to 21%, while that for the Web administration was equal to 11%. Analysis of the overall scale comprised of 35 items for the entire group (N = 197) revealed an internal consistency measure (i.e., Cronbach’s index of internal consistency; also known as the alpha coefficient, or a) equal to .57, indicating a somewhat less than desirable degree of reliability. The internal consistency measure for the “paper” sample (a = .56) was nearly identical to that for the Web sample (a = .59).

On average, teachers who responded to the Web-based version answered slightly less than 22 out of 35 items correctly. Out of the seven competency areas, as delineated by The Standards, the highest overall performance for this group was found for Standard 3—Administering, Scoring, and Interpreting the Results of Assessments (M = 3.95; maximum possible score = 5). The lowest performance was found for Standard 5—Developing Valid Grading Procedure (M = 2.15).

Similarly, teachers who responded to the paper version answered slightly more than 22 out of 35 items correctly. Identical to the performance of teachers in the Web group, these teachers exhibited their highest overall performance on Standard 3—Administering, Scoring, and Interpreting the Results of Assessments (M = 3.96; maximum possible score = 5). The lowest performance was also found for Standard 5—Developing Valid Grading Procedure (M = 2.02).

Scores for each of the seven Standards as well as the total scores for the two groups of teachers were compared by conducting independent-samples t-tests (a = .05). Examination of the results revealed no significant differences between the two groups for scores on any of the seven Standards, nor for the total scores. The results of all t-tests are presented in Table 1.

 Research Question 2:  What patterns of survey nonresponse result from the administration of the same survey?

A potentially more interesting set of findings resulted from the originally unintended follow-up survey conducted with those teachers in the Web group who chose not to complete the survey of primary interest in this study. These teachers (N = 457) were emailed and asked to indicate from seven options the main reason they did not respond to the survey. Ninety-four teachers, representing 21% of the nonrespondent group responded to the follow-up (interestingly, this was nearly twice the number that responded to the original survey). The frequencies and percentages of response to the seven options are presented in Table 2. The teachers were instructed to explain their response if they marked option #7 (“Other”). It is important to note that several teachers marked “Other,” but the explanations they provided clearly fit into one of the other categories. Additionally, many teachers provided explanatory comments, regardless of the option they selected. Examining the frequencies from Table 2, the reasons for nonresponse can be rank ordered as follows:

1)     I simply didn’t want to take the time to respond.
2)     I couldn’t access the survey due to limitations of technology.
3)     The survey was too lengthy.
4)     The topic didn’t interest me.
5)     I couldn’t access the survey due to my lack of technological expertise.
6)     I was afraid of the security/confidentiality of my responses.
7)     Other (The survey was too difficult.)

The most common reason for nonresponse to the original survey was “I simply didn’t want to take the time to respond,” with nearly one-half (48%) of nonrespondents indicating that this was their reason. Many teachers indicated that their schedules at school are simply too busy to permit them to do any “extra” things:

I didn’t have the time, no matter how short or long.

I simply do not have the time to do this at this time. I could maybe do it this summer.

I looked at it before school and thought that I would do it after school. However, I forgot to do it simply because of how busy I have been with school.

My day is packed FULL with virtually no time to spare. It’s not that I didn’t WANT to take the time… other necessary tasks had absolute priority on my time… and if you have ever taught a core high school subject for any length of time, you’d know that.

I started doing the survey and it was taking longer that I cared to spend doing the survey. I stopped doing the survey after about 3–4 questions.

Importantly for this paper, a considerable number of teachers (20%) also indicated that they did not respond due to technological limitations.: Several teachers indicated that they were not able to access the survey, that they completed the survey but it “disappeared” when they attempted to send it, that they tried to complete the survey over the course of one or two days and lost their previous answers, or that their email provider somehow altered the survey’s URL:

I am sorry I didn’t reply, but I am not able to open the survey.

I couldn’t reply. You had a link to click on and nothing happened when I clicked on it. Also, I couldn’t put an X in the spot you asked me to on this email.

I didn’t fill out the survey because the hyperlink did not connect me. I didn’t want to take the time to fill in the address.

Our set-up at [our school] is VERY limited. We cannot open the document – I tried, but the computer would not allow this to happen.

I did not respond to you survey because I could not get into your site.

I was unable to respond to your survey because of poor school equipment.

Began answering the questions. When my planning period was over, I stopped answering. When I came back the next day, none of my answers had been saved. I simply do not have the time to spend two planning periods answering a survey.

… I am the tech rep for my building… three other teachers in the building sought my help – two of them had the same trouble I did – it disappeared without sending upon completion. The third teacher responded to approximately 1/2 and attempted to save her responses so she could complete it at another date. When she went back to it, all of her answers were gone.

 I started but was interrupted and accidentally closed my browser, erasing what I had previously written.

 Since most of the staff use a Webmail interface supplied by [private computer service provider], I suspect that it may have truncated the URL incorrectly making it incomplete.

Fourteen teachers (15%) indicated that the survey was simply too long to respond to. Many of those who checked this option also checked the option stating that they did not want to take the time to respond.

Four teachers (4%) indicated that they could not access the survey due to their own lack of technological expertise. Their explanations included the fact that they accidentally deleted the original email message, or did not know how to access the survey if the hyperlink did not seem to work:

Deleted it accidentally.

I deleted the message accidentally and don't know where the survey is.  Sorry, a slip of the key.

I was one of the people who did respond to your survey.  However, I could not get the survey by clicking on your link.  I had to forward your message to our computer technician who pasted someway for me to respond.  Perhaps this is why you got a poor response.  Maybe your link didn't work for other people and they just gave up.

Finally, two teachers who checked “Other” provided explanations that could not be categorized into any of the other options. These teachers offered a variety of explanations for their selection. These teachers indicated that they could not understand the nature of the questions appearing on the survey:

I felt that the survey was extremely confusing.

I have to tell you that I did muddle through the first 20 or so questions and then quit.  Maybe it's because my background isn't in testing, but your wording of the questions and answers was incredibly difficult. 

Discussion

Research has begun to demonstrate that utilizing a Web-based approach is a viable means of gathering survey data. However, further research must be conducted on this mode of delivery in order to pass judgment on its relative merits. There are several advantages to electronic surveys, in general. These include such things as cost and time savings, as well as the ease of transfer of responses into a database. Are these advantages over paper-and-pencil surveys great enough to “counteract” the problems associated with nonrandom samples? Are potential respondents comfortable enough with technology to respond to surveys online? Or do they remain apprehensive about potentially realistic concerns such as anonymity, confidentiality, and security of their provided information? If the answer to the latter question is “yes”—which certainly seemed to be the case in the present study—do we as researchers end up with a “differential” type of random sample resulting from respondent self-selection? This, of course, is a concern for any type of survey research, but can we be sure that we are not ending up with a sample whose characteristics are different from those realized through paper-and-pencil forms of surveys? To address these and related issues, further research is most certainly called for.

A few interesting methodological issues related to the Web-based mode of survey delivery were raised as a result of this study. First, the researcher, working in consultation with an information technology professional at the institution, checked the viability of the Web-based survey on a variety of Web browsers (e.g., Internet Explorer, Netscape Navigator, and others), as well as various versions of those browsers. Although these various combinations of browsers and versions were verified, some teachers still were unable to access the survey, as evidenced by their comments in response to the follow-up survey. This may have been due—at least in part—to older hardware and/or software in the schools.

Second, it is important to note that many of the teachers surveyed in this study worked in rural school settings. Many of those districts contract with private Internet service providers (ISPs) to provide Internet access and email services. Another possible cause for teachers’ inabilities to access the survey may have been caused by the various settings provided by the ISPs who supply service to these districts. Following the initial email message (i.e., cover letter) for the primary survey, several teachers corresponded with the researcher to inform him that their district did not support hyperlinks embedded in email messages. In some instances, the URL appeared in the message, and when teachers clicked on it, nothing happened (of course, this is easily remedied by copying and pasting the link into the URL bar of a browser window). In other situations, the URL may have been truncated or perhaps did not even appear in the body of the email message. Still other teachers informed the researcher that their district did not permit delivery of email messages from unknown sources or access to unknown URLs. It is possible then that for some teachers access to the survey was literally impossible (and, therefore, participation was not a conscious decision made on the part of individual teachers), or that some of them never even received the email cover letter. Obviously, this would have had a substantial impact on the ultimate rate of response to the survey.

Third, and somewhat related to the second issue above, involves the issue of technology literacy. Although we have a tendency to believe that everyone—especially in the educational community—in this day and age is technologically literate (e.g., knows all about browsers, how to maneuver around the World Wide Web, how to alter email settings, etc.), the fact remains that many individuals are simply not literate in this sense. Numerous potential respondents to this survey, when encountered with a hyperlink that did not “work,” were apparently unaware that the URL could be copied and pasted into a browser window; they simply gave up and did not try further to access the survey. Also troublesome to the researcher was the fact that some teachers began to respond to the survey, logged off of their computers, logged back onto them the next day, and expected to gain access to their partially completed survey, ready to finish and submit. The vast majority of Web pages do not work this way.

Recommendations

Based on these identified methodological issues, several recommendations are offered. These recommendations address the broad issue of logistically creating a viable system of data collection for Web-based surveys, with the intention of maximizing the rate of response.

Recommendation #1:  Web-based surveys—or any other type of electronic survey, for that matter—must be subjected to extensive pilot-testing, above and beyond the degree to which traditional surveys are tested. Accessibility to the survey must be established as the result of thorough testing of the instrument on all possible types of browsers (i.e., Netscape, Internet Explorer, Safari, etc.), platforms (i.e., Mac, PC-compatible), networks, and email programs (if cover messages containing hyperlinks to the survey will be utilized) as well as tested on every conceivable combination of the aforementioned hardware and software. Although this is especially important in technology-challenged school districts, it is also critical in those districts that are not. This type of pilot testing was attempted in the study at hand, but admittedly it was somewhat of a haphazard effort. We tested the instrument on “popular” browsers and only on the more recent (i.e., within the past four or five years) versions of those browsers, thinking that most districts would have these combinations. It is also due to this recommendation that a greater level of responsibility must be placed on the researcher to be as informed as possible about the population of interest and the associated level of accessibility to the survey. This may require the researcher to contact districts where potential respondents are employed and speak to their technology coordinators in order to find out what browsers, versions of browsers, platforms, network software, and email programs they are using. The survey should then be tested on one or more computers in each district prior to the administration of the actual survey to potential respondents.

Recommendation #2:  In order for the use of Web-based surveys to be a comparable methodology, respondents must be able to essentially respond to them in the same manner as they would a traditional pencil-and-paper survey. This implies that when completing the survey, they must be afforded the opportunity to pause, save their work, and complete the survey at a later time. This would likely require that each potential respondent be given his or her own unique password or identification number in order to gain initial access to the survey, save intermediate responses, and return to complete the survey at a later time. In the study at hand, several teachers indicated that when they encountered this problem of not being able to resume their completion of the survey, they simply opted not to do it at all largely due to the amount of time that starting over from Question #1 would require.

Recommendation #3:  Researchers must be prepared for individuals who are technologically challenged. Several teachers in this study indicated that they wanted to complete the survey, but could not gain access. Alternatives should be provided for the potential respondents. One alternative might be a PDF version of the instrument that could be printed, completed, and then faxed or mailed back to the researcher. Another option could be the survey in a template or “forms” version, created in Microsoft Word or other word processing program. This would permit the respondent to download the survey, complete it electronically, re-save it, and then email it back to the researcher. This completed version could also be printed and faxed or mailed back. Utilizing all of these response options would undoubtedly improve the response to the survey.

Recommendation #4:  Finally, researchers may also want to contact the ISPs for smaller, rural school districts to collaboratively work with them in order to investigate how—or even if—these problems of access can be circumvented. Numerous teachers in the study seemed to believe that their inability to gain access to the survey was a direct result of the email and networking software being used by their ISPs.

Generally speaking, further research on the feasibility of the Web-based delivery of surveys is most certainly warranted. In addition, as a result of this study, other vitally important methodological issues have had some light shed upon them. The issue of equal access to Web-based surveys—and to electronic surveys, in general—as well as the issues of the technological capabilities of potential respondents and access to a survey in their workplace truly calls into question the extent to which educators, especially in K–12 settings, should be surveyed via electronic means. At this point in time, if the educational research community continues to do so—without heeding the recommendations offered here—we can only assume that we are, in all probability, obtaining anything but a representative sample.

References

American Federation of Teachers (AFT), National Council on Measurement in Education (NCME), & National Education Association (NEA) (1990). Standards for Teacher Competence in the Educational Assessment of Students. Retrieved April 21, 2003, from http://www.unl.edu/buros/article3.html

Brookhart, S. M. (2001). The Standards and classroom assessment research. Paper presented at the annual meeting of the American Association of Colleges for Teacher Education, Dallas, TX. (ERIC Document Reproduction Service No. ED451189) 

Carbonaro, M. & Bainbridge, J. (2000). Design and development of a process for Web-based survey research. The Alberta Journal of Educational Research, 46, 392-394.

Dillman, D. A. (1998). Mail and other self-administered surveys in the 21^st century: The beginning of a new era. Unpublished manuscript, Washington State University, Pullman, Washington.

Dillman, D. A. & Bowker, D. K. (2001). The Web questionnaire challenge to survey methodologists. In U. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science. Lengerich, Germany: Pabst Publishers.

Heflich, D. A. & Rice, M. L. (1999, March). Online survey research: A venue for reflective conversation and professional development. Paper presented at the Society for Information Technology & Teacher Education International Conference, San Antonio, TX. (ERIC Document Reproduction Service No. ED432283)

Idleman, L.  (2003, April). Comparing responses to mail and Web-based surveys. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Matz, C. M. (1999). Administration of Web versus paper surveys: Mode effects and response rates (Master’s thesis, University of North Carolina). (ERIC Document Reproduction Service No. ED439694)

Mertler, C. A. (2001, October). Lessons learned from the administration of a Web-based survey…. Paper presented at the annual meeting of the Mid-Western Educational Research Association, Chicago, Illinois.

Mertler, C. A. (2002a, April). Web-based surveys: Guiding lessons for their use. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, Louisiana.

Mertler, C. A. (2002b). Demonstrating the potential for Web-based survey methodology with a case study. American Secondary Education, 30, 49-61.

Mertler, C. A. & Earley, M. A.  (2003, April). A comparison of the psychometric qualities of surveys administered by Web and traditional methods. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Montez, J.  (2003, April). Web surveys as a source of nonresponse explication. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Plake, B. S. (1993). Teacher assessment literacy: Teachers’ competencies in the educational assessment of students. Mid-Western Educational Researcher, 6(2), 21–27.

Saphore, R. B. (1999). A psychometric comparison of an electronic and classical survey instrument (Doctoral dissertation, University of Alabama, 1999). Dissertation Abstracts International, 60, 3976.

Schillewaert, N., Langerak, F., & Duhamel, T. (1998). Non-probability sampling for WWW surveys: A comparison of methods. Journal of the Market Research Society, 40, 307-322.

Solomon, D. J. (2001). Conducting web-based surveys. Practical Assessment, Research, and Evaluation, 7(19). Available online: http://PAREonline.net/getvn.asp?v=7&n=19.

Author

Craig A. Mertler, mertler@bgnet.bgsu.edu
Division of Educational Foundations & Inquiry
School of Educational Leadership & Policy Studies
College of Education & Human Development
Bowling Green State University
Bowling Green, OH 43403