Measuring Online Experience: It’s About More Than Time!

By Bonnie Lida Rogers

“Experience … the fact or state of having been affected by or gained knowledge through direct observation or participation” (Merriam-Webster, 2003).

Most researchers studying online behavior use “experience” level as a way to categorize individual differences in using the Internet. For the majority of these studies, however, experience level has been defined by time and/or frequency measurements, such as “How long have you been using the internet?”; “How much time do you spend online?”; or “How often do you use the Internet?” Two recent longitudinal studies, which report the changing online population in the United States (Cole, Suman, Schramm, Lunn & Aquino, 2003; Lenhart, Horrigan, Rainie, Allen, Boyce, Madden, & O’Grady, 2003), show that the amount of internet usage is leveling off, but that the breadth of demographic user characteristics are continuing to expand. Therefore, measuring experience level by frequency of use and longevity of use may not be truly representative. Although measures of time and frequency typically show a high correlation, the online user’s experience level cannot be adequately evaluated by considering it as a single dimension. A more descriptive measure of experience might be “how” users experience the Internet rather than “how often.”

People use the Internet for many different purposes. Computer game enthusiasts compete online and communicate with friends using Instant Messaging. Stock brokers research and monitor online stock prices and trading. Many of us pay bills and bank online, communicate via e-mail, and browse/shop online. It is also possible that each of these users could report spending roughly the same amount of time online (e.g., 4 hours a day). By most standards, all of these users would be considered “experienced” or “expert” online users. However, their specific online activities, their knowledge of computers and the Internet, and how they feel about being online could differ significantly. In order to obtain a more robust measure, researchers may want to consider how and what users think, feel, and do online when they assess level of experience. In this article we explore each of these areas and present a new measure of measuring online experience.

METHOD

Three components of experience were considered in the development of the Internet Experience Scale: 1) Conative (what users “do” online) by measuring the user’s time, frequency, and activities online; 2) Cognitive (what the user “thinks” online) by measuring the user’s technical knowledge of computers and the Internet, or their “online thinking”; and 3) Affect (what the user “feels” online) by measuring their feelings and attitudes while online. A pool of questions to comprehensively assess each of these areas was collected, along with descriptive demographic and environmental queries. The questionnaire was administered to 168 graduate and undergraduate business and psychology students at Wichita State University. The following is a partial analysis of the results looking at affect and activities in the first iteration.

Our sample population consisted of 168 students – 93% full time, with 81% in undergraduate and 19% in post-graduate programs. The mean age was 24 years, with a range of 18 to 55. The gender of the sample was 35% male and 65% female. Ninety-three percent of the sample reported accessing the Internet from their home. Forty-six percent reported using dial-up for their Internet connection, while 43% used DSL/Cable; and 3% did not know.

RESULTS

Conative Measures

When queried about their online usage and activities, 91% of participants reported being online that day or the previous day. Participants were asked to respond to a list of 16 various online activities on a Likert-type 5-point scale (with anchors of “hourly” and “never”), as well as time online, frequency online, and online longevity queries. Ninety-seven percent of participants reported they were online for less than four hours in their last online session. Figure 1 shows the typical daily usage. Ninety-eight percent reported being online at least a few times per week, and over 65% have been online over 5 years.


Figure 1. Reported daily online usage.

Figure 1. Reported daily online usage.

Factor Analysis of Internet Activity Variables

Principal factors extraction with Promax rotation was performed using SPSS on the 16 activity items for the sample of 168. Results revealed three main factors (only activities loading .5 or above were retained). The results, including the interpreted factor labels, are listed below.

Table 1. Factor Analysis of Internet Activity Variables

FACTOR 1: EXPLORING
LOADING
1. Searched for information using a search engine (such as Google, Lycos, MSN) .761
2. Browse the Internet for fun .740

3. Compare information or prices using a comparison website (webbot)
.672
4. E-mail .616
5. Play a game .542
FACTOR 2: FINANCIAL
1. Pay bills .739
2. Bank online .702
3. Buy something at an online auction (such as eBay or Ubid) .601
4. Buy a product from an online store .595
FACTOR 3: SOCIAL
1. Participate in chat room .823
2. Listen to or download online music .603
3. Join a newsgroup .524
4. Instant messaging .511

Internet Activity Scale Reliability Analysis

In order to assess the internal consistency of the activity scale using the 13 activity variables from the factor analysis, the data were analyzed using scale reliability analysis in SPSS. A Chronbach’s Alpha of .74 was calculated reflecting adequate internal consistency of the items.

Reported Internet Activity Levels

This 13-item activity scale was sum-scored and used to divide the total pool of subjects into three groups – high activity, moderate activity, and low activity. Activity level is a score reflecting the number of different activities and the frequencies with which the participants engaged in them. Analysis found the mean score on the scale to be 33.7, within a range of scores from 16 to 58. Of the 168 cases, 60 (35.7%) were categorized as high level of activity (range 37-58), 52 (31%) moderate level of activity (range 31-36), and 56 (33.3%) were grouped as low level of activity (range 16 – 30).

Affective Measures

Participants were asked to respond to a list of 25 questions regarding their feelings about being online on a Likert-type 6-point scale (with anchors of “strongly disagree” and “strongly agree”). Questions were adapted from validated online surveys (Hoffman & Novak, 1997; GVU, 1997), as well as original queries.

Factor Analysis of Internet Affect Variables

Principal factors extraction with Promax rotation was performed using SPSS on the 25 affect items for the sample of 168. Results revealed four main factors (only activities loading .5 or above were retained.) The results, including the interpretative factor labels, are listed below.

Table 2. Factor Analysis of Internet Affect Variables

Factor 1: CONFIDENCE
1. I am in complete control when I use the Internet. .817
2. I consider myself knowledgeable about good search techniques on the Internet. .804
3. I am extremely skilled at using the Internet. .801
4. If I had a problem using the Internet, I could solve it one way or another. .784
5. I would prefer to learn how to use the Internet on my own. .691
FACTOR 2: INVOLVEMENT
1. I feel upset when my Internet access is not working. .794
2. I look forward to receiving e-mail messages. .747
3. I think of the Internet as a tool to make life easier. .684
4. Time seems to go by very quickly when I use the Internet. .584
FACTOR 3: NOVELTY
1. I enjoy visiting unfamiliar Internet sites just for the sake of variety. .862
2. I enjoy exploring the Internet. .845
3. I look forward to checking out new information on the Internet. .840
FACTOR 4: FLOW
1. I experience a sense of achievement when I use the Internet. .783
2. I forget about my immediate surroundings when I use the Internet. .782

Internet Affective Scale Reliability Analysis

In order to assess the internal consistency of the revised affect scale, the data in this sample were analyzed using scale reliability analysis with SPSS. A Chronbach’s Alpha of .84 was calculated, reflecting good internal consistency of the items.

Discriminant Function Analysis

To determine if the users’ online activity level can be predicted by online affective measures, a direct Discriminant Function Analysis (DFA) was performed using the four factor scores calculated on the 14 affect variables as predictors of membership in the upper and lower (high and low) activity levels. To differentiate activity level more completely, the moderate activity level was not included in the analysis.

A DFA was performed on 115 cases, 59 classified as high activity level and 56 as low activity level (see Conative measures). One canonical function was used, and produced an eigenvalue of .452, canonical correlation .558. A Wilks’ Lambda of .689 was produced, indicating that there was significant discrimination between the two groups of activity levels. The four affect factors correctly classified 75.7% of the original grouped cases.

DISCUSSION

Although there is a need to sample a more diverse audience to improve the generalizability of this measurement scale, the university population does offer a representative sample of the demographic profile of the Internet user. The most recent PEW survey of Internet usage (Lenhart et al. 2003) lists the Internet user as: younger, employed, white, well-educated, suburban/urban, and parents of children living at home. This sample tapped into at least four of those demographic descriptors. A second iteration of this scale will broaden the sample to older adults, parents, and more affluent audiences.

Our initial analysis suggests that what people are doing online (activities) and how they feel (affect) while online is a more robust measure of their online experience level than simply the time and frequency of going online. Revisions of this instrument will allow us to more fully understand the conative and affective measures, as well as develop a cognitive measure to assess experience.

REFERENCES

Cole, J. I., Suman, M., Schramm, P., Lunn, R. & Aquino, J. (2003, February). Surveying the digital future – Year three. Los Angeles, CA: University of California Los Angeles, UCLA Center for Communication Policy. Retrieved May 31, 2003, from http://www.ccp.ucla.edu/

“experience.” Merriam-Webster Online Dictionary. 2003. Retrieved June 15, 2003, from http://www.merriam-webster.com/

GVU (1997). 8th WWW User Survey, Graphic, Visualization, and Usability Center (GVU) at the Georgia Institute of Technology. Available from http://elab.vanderbilt.edu/research/data/gvusurvey/project2000.gvu8.html

Lenhart, M., Horrigan, J., Rainie, L., Allen, K., Boyce, A., Madden, M. & O’Grady, E. (2003, April). The ever-shifting internet population: A new look at internet access and the digital divide. Washington, DC: The PEW Internet & American Life Project. Retrieved May 30, 2003, from http://www.pewinternet.org/

Novak, T. & Hoffman, D. (1997). Measuring the flow experience among web users. Presented at Interval Research Corporation on July 31, 1997. Retrieved June 1, 2003, from http://elab.vanderbilt.edu/research/manuscripts/index.htm