Impressions and Recommendations

Activities and Findings

End-User Feedback

Feedback from end-users of the service was anecdotal and self-reported. Often end-users would express their opinions about the value and usefulness of the service in email messages or at the conclusion of an interactive online reference session.

Generally the end-users of InfoEyes were very satisfied with and excited by this new service. They appreciated an online service that was designed specifically for them. Several InfoEyes patrons became frequent repeat customers of the service. They also were ardent supporters.

Accessibility

The accessibility challenges associated with the InfoEyes service were varied. They were not completed resolved during the beta phase. The text chat aspect of the InfoEyes service used an interface that refreshed the screen every 15 seconds. This created accessibility challenges for individuals who use screen reader software, such as JAWS for Windows and WindowEyes. When a software program or website has an embedded automatic screen refresh capability, the refreshed screen often is interpreted by screen reader software programs as an entirely new screen. The screen reader software program will begin reading aloud the contents of the screen each time it refreshes.

We also experienced some accessibility challenges with the various enhanced reference service options we tested. For example, the text chat window within QuestionPoint Enhanced Communications was difficult or impossible to read with screen reader software when the window was in its default "docked" position within the overall QPE interface. The simple workaround to this accessibility challenge was to ask each reference librarian on duty to "undock" the text chat window whenever he or she answered a request to initiate a QPE interaction. A blind programmer also was hired to develop scripts so that JAWS would work with QuestionPoint Enhanced. This solution, however, required a blind end-user to now download both the QPE end-user client and the set of JAWS scripts before he or she could even ask a question.

The voice-over-IP aspects of the various software options we tested did not in-and-of-itself present major accessibility challenges. Most InfoEyes patrons appreciated being able to hear the reference librarian's voice coming over the Internet, even if the patron did not have a microphone to speak back. Although it initially seemed counterintuitive, enhanced virtual reference interactions in which the reference librarian was communicating primarily via VoIP and the patron primarily via text chat seemed to work well and to the satisfaction of both parties.

Satisfaction with the Service

Patrons who used the InfoEyes service during the beta test period were generally satisfied with the service. Satisfaction parameters included the thoroughness of the answers, the quality of the answers, and, for email questions, the turnaround time.

Value of the Service

The users who tried InfoEyes during the beta testing phase and provided feedback tended to highly value the service. For many individuals, InfoEyes was the first online reference and instruction service designed specifically to meet their needs. Some individuals specifically praised the value of the instructional sessions, because it helped them become independent seekers and users of information resources on the Internet and the Web.

Software

QuestionPoint email

The email reference module within QuestionPoint worked quite well. Not only was it easy for patrons to pose questions and for InfoEyes service providers to respond, but it also was very easy to manage the overall flow of email questions. InfoEyes service providers could easy monitor the progress of responding to email questions.

QuestionPoint basic text chat

From the perspective of the InfoEyes project, the basic drawback of the basic text chat module of QuestionPoint is that the screen automatically refreshes every 15 seconds. For a blind or visually impaired individual using screen reader software, such as JAWS for Windows or WindowEyes, each screen refresh is interpreted as a new screen to be read. This was very annoying to users.

QuestionPoint Enhanced Communications

After trying QuestionPoint Enhanced for several weeks, the InfoEyes Advisory Council decided to suspend active testing of this module with the InfoEyes service population. The VoIP functionality performed inconsistently, often with long transmission delays. The text chat box had to be maximized and undocked by the reference librarian in order to improve the accessibility of the contents of the text chat for end-users using screen reader software. The end-users had to download a plug-in to enable VoIP, co-browsing, and application sharing.

iVocalize for Enhanced Reference Service

As with QuestionPoint Enhanced Communcations, with iVocalize end-users are required to download a plug-in in order to access the room. Downloading the iVocalize plug-in, however, was fairly transparent to the user and generally easy to implement. Although the process of downloading the iVocalize plug-in was very easy, in a few instances patrons expressed problems or concerns about having to download a plug-in. Downloading the QuestionPoint enhanced end-user client was a more elaborate process, requiring a user-initiated software installation.

The iVocalize software was very accessible to blind and visually impaired users. The VoIP function worked well and reliably, with little delay time. Patrons seemed to appreciate the VoIP function even if they did not have a microphone at their end and had to resort to using text chat to communicate. The ability to co-browse with patrons throughout the web, even into proprietary databases, was very useful in numerous reference and online instruction interactions. The software can be configured so that the URLs of all webpages visited are automatically added to the text chat transcript. The transcripts are easily saved.

It is possible to record the audio portion of any interaction in an iVocalize online room. However, the InfoEyes Advisory Council decided not to use this feature of the system, because we wanted to protect patron privacy and confidentiality. To further ensure patron confidentiality, we set up the InfoEyes iVocalize room so that only two people the reference librarian and the patron could be in the room at any one time.

The fact that iVocalize did not directly interface with the QuestionPoint system caused some service and administrative challenges. Because the iVocalize software is not designed specifically for providing reference service, it also lacked some features commonly found in VR software systems, such as the ability to create and easily insert scripted messages.

iVocalize for online end-user group training sessions

The iVocalize software worked very well for online end-user group training sessions. The instructor was able to co-browse into password-protected databases for the purposes of demonstration and instruction. Much peer-to-peer sharing of tips occurred during these sessions, too.

iVocalize for online InfoEyes service provider group training sessions

The iVocalize software from Talking Communities also was used to provide group training sessions for InfoEyes service providers. The software proved very useful for this purpose.

iVocalize for meetings of the Advisory Council

All of the meetings of the InfoEyes Advisory Council were held in a large iVocalize online meeting room. Members of the Advisory Council included sighted, partially sighted, and blind individuals. The software worked well for these meetings.

Administration

Orientation and Training for InfoEyes Service Providers

Numerous online orientation and training sessions were provided for InfoEyes service providers by OCLC and the Core Project Team. These sessions were well-attended by staff members from the participating libraries. The mentor system that assigned a member of the Core Project Team to each participating library seemed to facilitate communication and encouraged continued commitment to the development, testing, and deployment of the InfoEyes service.

Technical Support

Providing technical support to participating libraries and patrons was a joint effort of OCLC, Talking Communities, and the Core Project Team. Other than the core accessibility challenges outlined above, generally technical issues were resolved quickly and satisfactorily via this team approach.

Scheduling

Creating a desk schedule involving so many states and time zones was a complex, arduous task, but overall the scheduling process worked well. Time zone differences and the fact that one participating state did not observe daylight savings time created some confusion. The process of resolving last minute reference desk substitutions was not as efficient as the overall scheduling process.

Funding and Seeking Funding

During the beta testing period the Core Project Team sought external funding for InfoEyes by applying for a major grant from a federal agency. Shortly after the conclusion of the beta phase of InfoEyes we learned that our application had not been funded.

Governance

Advisory Council

The governance structure and process have worked quite well, and they already have received some substantial tests. For example, the decision to switch from the QuestionPoint Enhanced Communications software platform to iVocalize as the basis for the enhanced InfoEyes service was not a trivial decision. The decision to drop the basic text chat option also was a substantial decision. The failure to attract external funding on the first attempt could have severely damaged or killed the project, but the participating libraries, working within the existing governance and management structures, was able to determine a way to sustain InfoEyes on a minimal budget and with a well-distributed in-kind contributions plan.

Reports

Monthly Statistical Reports

Statistical reports about the activity in the InfoEyes reference service, including the use of the free trial FirstSearch databases, were compiled and distributed each month. The cumulative report for the first six months of activity is contained in Appendix A. The Advisory Council initially decided that the standard monthly reports provided by QuestionPoint were not sufficiently detailed for the purposes of a beta test, so each reference transaction during the beta test period of reviewed and coded appropriately.

Website

Design

Already in 2004 the program website (www.infoeyes.org) has gone through several major revisions.

Maintenance

Maintaining the website has been difficult. More attention needs to be paid to ensuring the out-of-date information is removed and that all current information is accurate.

Usage Statistics

Summary data about the usage of the InfoEyes website can be found at http://www.infoeyes.org/stats.

Usage

Note: Please see Appendix A for a complete summary of the usage of the InfoEyes service during the beta testing phase.

Definitions

Total Sessions: The sum of all the discrete sessions recorded in the set of closed interactions for the month being reported.

Countable Sessions: Two situations caused some sessions not to be counted:

If two or more sessions in close temporal proximity to each other were clearly from the same patron with the same information need, the cluster of sessions was counted as only one session.

If a session clearly displayed a failure to communicate (e.g., because of basic system problems, or the patron never responded and expressed the need), it was not counted as a session.

Real Sessions: Sessions that appeared to involve patrons with real information needs.

Interest Sessions: Sessions that appeared to be from non-InfoEyes team members who were more interested in just seeing or testing the software or service than in satisfying a real information need. Often these visitors were professional colleagues from other libraries.

Practice Sessions: Sessions that appeared to be test and practice sessions between two or more InfoEyes team members.

Timestamping: Sessions are timestamped when they commence. In the case of email questions, that means the time they are submitted, not when they are answered by an InfoEyes reference provider. The timestamp is GMT (Greenwich Mean Time) minus six hours. Evidently, no adjustments are made to the timestamp when daylight savings time commences in April and concludes in October.

Note: The InfoEyes service had a soft launch on March 1, 2004, but activity did not really pick up until March 17, and the enhanced communication software was not utilized until March 22. On April 29 public access to the QuestionPoint Enhanced mode was discontinued. It was replaced on June 7, 2004 with iVocalize software from Talking Communities. Thus during the month of May 2004 patrons had no access to an enhanced mode of communication, only email and basic text chat.

Overall Volume of Use of InfoEyes

A total of 295 total sessions occurred during the first six months of the InfoEyes service. Of those, 233 were defined as countable sessions, using the criteria outlined above. Of the 184 days during the six-month period, 102 (55.4 percent) contained at least one InfoEyes session. The average number of countable sessions per active day for the six-month period was 2.28. However, there was a noticeable downward monthly trend in the average number of countable sessions per active day, starting in May with 5.14 and concluding in August with 1.17.

Of the 233 countable sessions, 161 (69.1 percent) were real (i.e., based on the logs, these appeared to be real information needs expressed by real patrons), 25 (10.7 percent) were interest sessions (i.e., the person was interested mainly in learning more about InfoEyes), 41 (17.6 percent) were practice sessions involving two or more InfoEyes service providers, and 6 countable sessions (2.58 percent) were of unknown basic intention.

The day of the week during which the 233 countable sessions were initiated was fairly evenly spread across the seven days of the week. Thursday was the most active day of the week, during which 60 (25.75 percent) of the countable sessions were initiated, and Sunday was the least active day of the week, when only 7 (3 percent) sessions were initiated.

In terms of how the countable sessions spread out across the 24 hours of the day, there were two peaks of activity: from 10-11 in the morning, and from 3-4 in the afternoon. This pattern of a late morning and afternoon peak in demand appears to be fairly common among online library services, remote use of online catalogs, etc. These hourly data, however, must be interpreted with caution for several reasons. First, all session activity for the email and basic text chat sessions were time-stamped using a clock set to GMT (Greenwich Mean Time) minus six hours, even though the actual activity was spread at least across the four times zones in the contiguous 48 states. Also, as the project continues into the fall and winter months, evidently no change in the time stamp is made when Daylight Savings Time ends and commences. Nevertheless, it appears that demand for InfoEyes service was particularly light between 10 p.m. and 8 a.m., especially when one considers that 38 percent of all interactions were email questions.

Mix of email, text chat, and enhanced sessions

Of the 295 total sessions initiated during the first six months of the InfoEyes online reference service, 114 (38.64 percent) were enhanced sessions, 112 (37.97 percent) were email questions, and 69 (23.39 percent) were basic text chat sessions. If, however, the March and April usage data are removed, beginning in May and continuing through August the utilization of basic text chat (just 16 total sessions) and enhanced (just 7 total sessions) has been very low. The use of email as an interaction mode has remained much more consistent across the entire first six months of the service. Please bear in mind, however, that no enhanced online reference service communication mode was offered from April 29 until June 7 as the InfoEyes service providers prepared to switch from using QuestionPoint Enhanced to iVocalize from Talking Communities.

The average response time for all email questions was 17 and two-thirds hours. The average basic text chat session lasted a little over 9 minutes, and the average enhanced session lasted slightly more than 4 and one-half minutes.

Use and Usefulness of the FirstSearch databases

Of the 12 databases that were available for trial use, WorldCat (24 percent) received the most use, followed by ArticleFirst (21 percent) and Periodical Abstracts (20 percent). The trial passwords were shared both with InfoEyes service providers as well as with patrons who attended the group training sessions during the summer months.

Public Group Training and Orientation Sessions Through OPAL (Online Programming for All Libraries)

Several InfoEyes users reported that the group training and orientation sessions on Google and FirstSearch were fantastic. The sessions were held during June and July. Separate clusters of sessions covered introductory and intermediate Google searching, as well and introductory and intermediate FirstSearch searching. The sessions introduced small groups of users to the overall InfoEyes service, and actually created some activity with the one-on-one reference service.

Announcing, Promoting, and Reporting Out

InfoEyes was announced and promoted through messages to email discussion groups, blog postings, and printed and electronic newsletters distributed by the participating libraries. Several presentations at conferences, as well as one published article, served further to raise awareness about InfoEyes.

Presentations

Presentation at the KLAS (Keystone Library Automation System) Users Conference (April 2004)

Sussman, Diana Brawley, and Linda Rossman. Inside InfoEyes: A Multi-State Virtual Reference Adventure. A presentation made at the Users Conference of KLAS (Keystone Library Automation System), Boston, Massachusetts, April 23, 2004.

Presentation at NLS Biennial Conference in Rapid City, SD (May 2004)

Bell, Lori, Diana Brawley Sussman, Kim Charlson, Mary Mohr, and Jeff Penka. InfoEyes. A presentation made at the Biennial Conference of the National Library Service for the Blind and Physically Handicapped (NLS), Rapid City, South Dakota, May 5, 2004.

Presentation at the Annual Conference of the American Library Association in Orlando, Florida (June 2004)

Bell, Lori, and Tom Peters. InfoEyes: An Online Information Community for the Visually Impaired. A presentation made at the Annual Conference of the American Library Association, Orlando, Florida, June 26, 2004.

Presentation at the American Council for the Blind Conference in Alabama (July 2004)

This presentation was made by Lori Bell, Director of the Mid-Illinois Talking Book Center.

Presentation at the Virtual Reference Desk Annual Conference in Cincinnati, Ohio (November 2004)

Made by Lori Bell, Tom Peters, Lori Thornton, and Crystal Lentz. A link to the presentation slides can be found at the following URL: http://www.vrd2004.org/proceedings/presentation.cfm?PID=311

Articles

Article in Library Hi Tech News

Peters, Tom; Bell, Lori; Ruda, Sharon; and Sussman, Diana Brawley. 2004. InfoEyes: A Virtual Reference Service for the Visually Impaired. Library Hi Tech News 21 (6): 5-11.

Forthcoming article in American Libraries