User Stages in Recruitment Process (click for larger Flickr sizes)

Stage One: The Newbie

The Newbie has likely heard about your program or research area from someone (like an academic adviser) or something (like a push advertisement), but she is not exactly sure what it is. Questions like ”Should I go back to school?” and ”…or should I get a job in industry?” need to be answered. From a power perspective, The Newbie has very little. She is asking “What is this program and is it right for me?”

Stage Two: The Powerless Prospective

The Powerless Prospective is powerless because he doesn’t know if he’ll be admitted. He’s solidly in information seeking mode to find out what makes each program different. If he experienced stage one at a different university, this is the first time he’s heard of you. He’s searching Google for increasingly accurate and knowledgeable terminology and SEO is now very important. He’s weighing the reputation of your university and the cost and determining ROI. He might be considering other research/study areas and it’s quite possible that he’s still considering industry options.

Stage Three: The Applicant

The Applicant is sold on the research area and your university and she’s revisiting your site with application logistics and deadlines in mind. She’ll want to get a real sense of the environment, either in person or virtually. And if she’s an international student, she’s concern about assimilation. She’s looking to see how your program collaborates across the university and is focused on the curriculum. It’s possible that she’s doing the same for three to five other academic institutions.

Stage Four: The Powerful Prospective

The Powerful Prospective is desirable. He’s been accepted to your program and possibly others and has to decide where to matriculate. He’s revisiting questions he had when was The Newbie, except he’s seen it all before. What makes your school different, again? What’s my ROI? If you have new content for him – possibly firewalled – this could be an edge.

Stage Five: The Enrolled Newbie

The Enrolled Newbie is familiar with your website, but a newbie all over again. She’s looking for advice on moving and life in your city. She’s considering social and professional networking opportunities. Important to note and realize, she is already a potential recruiter for your program.

1. Turn off the phone (both office and cell) before the user arrives.
2. If you're using a mac, turn off active screen corners and Spirited Away!
3. iMovie cuts off at twenty-some-odd minutes and takes as many to transfer to DV, so if you're trying to be fiscally inventive with your "video recording" keep the session you'd like to record to less than 20 minutes and record in one fell swoop. Otherwise use a real video camera. This item has been updated.
4. Snapz Pro doesn't seem to do too well with hour-long captures. Next time, capture individual tasks in separate movies.
5. Asking questions that branch out of the subject of the website, "Let's say you're new to Durham... where would you go to find a good place to eat?" can offer a nice glimpse into a person's local infoCloud without being too probing.
6. Asking the same question in different ways can sometimes chip away at a default answer and lead to insight.
7. It is hard to avoid asking leading questions (the same goes with speaking for the user when repeating their answer).
8. I think it would be more beneficial to have a third-party conduct the interview (so that users don't hesitate to provide criticism).

[UPDATE: Quicktime Broadcaster solves this problem nicely. You can record for long periods (an hour and a half was no big deal) and the file encodes while recording, so you don’t have to wait for the encoding process afterwards.]

Note: The brainstorming in this post is specifically geared towards a higher-ed departmental website that I’m redesigning this summer. I am interviewing five graduate students because faculty consensus (in my previous experiment with web-based surveys) was that graduate students are a major audience for the site.

Interview Goals

First on the agenda is to develop a list of goals for the interviews. What do I want to know (…or change? …or fix?) that requires (or benefits intrinsically from) an in-person observation and/or dialogue?

Goal 1: Figure out how current graduate students use the website.

Do current grad students even visit the website or is it primarily a recruiting tool and once students arrive, they no longer look to the departmental website for information/news/events? Are they looking/needing information that is hard to find, out of date or simply missing? What other information sources do they use (printed materials, advisers, other students, the university’s graduate school)?

Goal 2: Figure out how prospective graduate students use the website.

When these current graduate students were researching graduate programs, what did they look for? Did the current website meet their needs? What kind of recruiting information was lacking? Are we representing the town/university/school/department accurately?

Goal 3: Streamline current navigation in sections where grad students are a primary audience.

Is the current navigation intuitive? How should it be organized and what words/phrases should be used? Starting from a blank page, how do graduate students find information? (Google vs. university website vs. school website vs. departmental website search vs. departmental website navigation vs. personal bookmarks vs. trial & error vs. memory)

Goal 4: Understand the aesthetic and technical preferences of graduate students.

How do graduate students scan text on a website? Are they comfortable with JavaScript/AJAX features? What kind of content is optimal for short paragraphs vs. bulleted lists vs. lengthy descriptions? What kind of imagery is the most appealing (people/students/faculty vs. research/equipment/facilities vs. informational graphics)? What color palettes do a)Duke; b) engineers; c) researching the environment prefer? Which browser/OS are they most comfortable using?

Interview Methods

I will utilize the following methods in attempting to answer the above questions and report back with which methods yielded the most insight.

Method: Open-ended interview questions.

I will sit with the interviewee and ask qualitative questions about their experience. I’ll take notes in addition to using recording devices for easy recall and future study. I will need to prepare my questions in script-form and practice them before running the test.

Method: Record user response (audio, video, screen capture, eye tracking).

I will video-tape the user with my iSight camera during the interview and any experiments/tasks. I will also run a screen capture program (If they prefer Mac, it’ll be Snapz Pro. If they prefer PC, I’m not sure what I’ll use yet. Suggestions?). I should develop a file naming/participant numbering system so that the screen captures and iSight MOV files are in sync. I’ll position a text file with the user’s name and the date on screen so that it’s captured for easy scanning.

Method: Walkthrough/talkthrough task(s).

I will ask the user to guide me in finding something on the internet. I plan to do this twice, once in a unrelated Google search (to assess how they find information in general) and once or twice in a related website activity (to assess how they find information in context).

Method: Eeny, meeny, miny, moe task(s).

The EMMM tasks are quantitative questions, like “Which one do you like better?” I need a program that delivers a slide (or slides) across monitors, so that I can put an option on each monitor. I wonder if it’s possible to command two slide presentations at once (perhaps one from Pages and one from Powerpoint?) There’s got to be a better way… Synergy could be useful.

Method: Card sorting task(s).

I will use card sorting to get a graduate students’ opinions on website navigation. In the excellent Card sorting: a definitive guide, Maurer and Warfel suggest between 30 and 100 cards. With this in mind, I’ll most likely limit my cards to graduate student specific areas (faculty, research, about, grads).

The point is this: Using the one-on-one interview, I will gain valuable insight and experience that cannot be obtained from web-based surveys. By preparing interview scripts and methods before hand , I’ll be a more capable and unbiased observer/interviewer. In the end, I hope to incorporate intuitive components that are attractive to the graduate student audience into my upcoming website redesign.

Next up for this series…

I will be reporting back with the results of my one-on-one interviews. I hope to have screen captures and user videos to share. I also plan to write about using Google Analytics to help make redesign decisions; creating personas based on analytics, interviews and surveys; and writing usability reports.

Re’cen•cy Bi•as n. is when a person assigns greater value to the most recent piece of information observed.

An example from Tufte follows:

…when this board is mentioned at a major site (slashdot, Arts & Letters Daily, Kottke, NYTimes), our logs will show, say, 2000 visits from that site the first day, a 1000 the second, and then, as the mention slips into a screen position reachable only by scrolling, 100 on the third day, and by 5 days, nothing. The quality of the thread didn’t change in 5 days, only its relative position on the referring site.

Recency bias is also known as recency effect, which is in direct opposition to primacy effect (placing a greater value on the first piece(s) of information a person observes). Another way to explain it is this example from Wikipedia:

…if a driver sees an equal total number of red cars as blue cars during a long journey, but there happens to be a glut of red cars at the end of the journey, he or she is likely to conclude that there were more red cars than blue cars throughout the drive.

The two effects in conjunction, referred to as serial position effect, are illustrated in the informational graph below:

This graph from Wikipedia shows how primacy and recency effect word recall.

When some sort of unrelated task is assigned after an experience or observation, like solving a math problem in your head, it can reduce the effect of recency bias. I assume this is because information that would encourage a recency bias is stored in the same area of the brain utilized to solve the math problem.

Connected, it seems, is the law of small numbers. Defined by Wikipedia, the law of small numbers is the tendency to extrapolate big conclusions from small samples. An example, from the same wikipedia page, is:

…sports fans assuming that a few excellent performances are proof of a player’s underlying ability… or … would be investors, who may assume that a mutual fund’s record over one year is a reliable indicator of the manager’s skill.

Also relevant are the assumptions of the Belief-Adjustment Model by Hogarth and Einhorn referenced in the article Effects of information sequence and irrelevant distractor information when using a computer-based decision aid by Reneau and Blanthorne:

1. The sequence in which information is presented affects judgments in a predictable manner.
2. The presence of irrelevant distractor information reduces judgment accuracy.
3. Neither information sequence nor irrelevant distractor information affect judgment confidence.

The point is this: Recency bias is a tool we can use to improve user comprehension and overall satisfaction. By considering the sequence of information (keeping in mind that many times the “home page” of your website is not the landing page of users.) and the (intentional?) placement of distractors, we can avoid common pitfalls in user judgments and decision making. And holy cow! Recency bias is but one of an entire list of cognitive biases.

I will share my process in an ongoing series entitled “Research for a Website Redesign” and I’ll be sure to follow up on those steps that worked particularly well and those that didn’t make the mark.

Why collect feedback via a web-based survey?

I wanted to give a certain user group a voice without giving them a lot of influence in the actual design process. I went with a web-based survey because:

1. It generates valuable feedback.
2. The user-group will feel like their opinion matters. (And it does.)
3. The survey gives a voice to an important user-group without giving them decision making control. (A nice alternative to design by committee.)
4. I have the opportunity to ask my users specific questions.
5. A large group of people can be polled in a short amount of time. (No appointments to make.)
6. It requires little commitment from the user. (The survey is on the website they are familiar with, is unobtrusive and all questions are optional.)

Creating a mechanism for collection.

I wanted a system that kept users on the website they were already familiar with and also gave a visual cue when feedback was given for a particular page. Here’s what I came up with (to be released in shareable format, once I clean it up and package it).

I created a beta version of the site (e.g. beta.domain.com) that had a clickable red bar with the instruction “Click to provide feedback for this page.” at the top of every page. When a user clicked on the red bar, a survey dropped down. When the user submitted the survey, I was sent an email with their feedback and the color of the bar changed to green. The idea here is that a user can go through the entire site using the navigation and at any time, they can easily tell if they’ve left feedback for a page already. (They also have the option to leave additional feedback on pages they’ve already submitted feedback.)

Technical Stuff: The system is written in PHP and uses a single-line include in the header file. It sets individual cookies for each page where feedback is given. Right now the file just sends an email, but I am considering upgrading it to a database so that tallying results is easier. The drawback, of course, is that installation would be more complex. I am open to hearing your preference if you are interested in downloading the code for your own website.

Asking the right questions.

Having a web-based survey allowed me to ask some questions that I didn’t know the answer to (or to confirm some things I knew all along). The questions in the survey are highly dependent on the reasons for the redesign. Many of my questions were geared towards gauging the purpose and placement of content and improving the maintainability of the pages, post launch. Here are the questions that I put on the survey, all were optional. [] for a checkbox and () for a radio button.

1. [] This page does not belong in this section.
2. [fill in the blank] is the primary audience for this page.
3. The purpose of this page is to:
   [] Recruit
   [] Promote Research
   [] Satisfy ABET
   [] Inform
   [] Perform a transaction
   [] No purpose/unnecessary
4. The content on this page should be updated this often:
   () Frequently (several times a semester)
   () Every Semester
   () Every Year
   () Never (the content is evergreen)
5. [fill in the blank] is missing from the page.
6. This page would be more appealing if [fill in the blank]
7. [fill in the blank] should “own” the content on this page.
8. Additional thoughts/comments (strengths, weaknesses, etc.)
   [fill in the blank]

Getting answers from your users.

I gave the beta url to a small committee of departmental faculty members first, with a deadline of two weeks to fill the survey out. After the initial run, I opened the survey up to all departmental faculty, with another two week turn around time. In a month’s time I had well-documented and mostly valuable feedback for every single page of the website. About 95% of that feedback came in the first two weeks, I only received a handful of additional comments from the faculty-at-large.

The point is this: A light-weight and integrated web-based survey is an easy way to get feedback from a large user group. It communicates that you care about your users opinions without relinquishing control over design decisions. In the end, you have well-organized and well-documented feedback to use as justification for your redesign decisions.

Next up for this series…

Tallying results for the second question “[fill in the blank] is the primary audience for this page.” led me to my next step. Faculty felt, overwhelmingly, that prospective and current students (both undergraduate and graduate) were the primary audiences site-wide.

My next step will be to survey those user groups about the current website, but with a different strategy. I’ll recruit several students (ideally five for each audience) and give them specific tasks (I will outline these later) to be performed in-person.

I’m also preparing an entry about using Google Analytics and mind-mapping software to redesign information architecture.