CMSC 434 | Introduction to Human-Computer Interaction | Fall 2009
Task Centered Design, Prototyping & Usability Testing
You begin your design by getting to know the intended users, their tasks, and the working context of their actions. Then you consider what the interface design should look like, basing your design on real people, tasks, and needs. You go out and identify anticipated users, talk to them about what tasks they are trying to do, and understand their work context. Because your initial designs will need refinement, you will have to identify potential usability problems by evaluating your design and by crafting new ones. This is called iterative design.
In this project, you begin by identifying tasks and constructing several low-fidelity prototypes:
- articulating good task descriptions and scenarios
- brainstorming to create several low-fidelity prototypes
- evaluating the prototypes through task-centered walkthroughs
- creating a single high-fidelity prototype to conduct a usability test
- revising your design to create a final version
The process of this project generates deliverables (see class schedule for due dates):
- User needs: Tasks, scenarios, references (~6-12 pages)
- First design (2-3 low-fidelity prototypes + description or tutorial ~6-12 pages)
- Task list and questionnaires (~2-3 pages)
- Usability test report (~6-12 pages)
- Final project (revised prototype + history ~15-25 pages)
The final result is a web based final report (revised prototype + history ~15-25 pages), for example see this set of student project final reports:
This semester we will focus on social networks. Some projects may be directed to participate in the CHI 2009 Student Design Competition:
You can pick any web-based project using any web technology as long as it has a strong social component. You should start thinking about possible projects by looking at the Term Projects page for ideas, and to post your own ideas to recruit other students to create a group.
- 1 Deliverables
- 1.1 5%: Proposal
- 1.2 10%: User needs: Tasks, scenarios, and references (~4-8 pages)
- 1.3 15%: First design (2-3 low-fidelity prototypes + description or tutorial ~6-12 pages)
- 1.4 10% : Task list and questionnaires (~2-3 pages)
- 1.5 20%: Usability test report (high-fidelity prototype + test ~6-12 pages)
- 1.6 20%: Final report (revised prototype + description)
- 1.7 Submission plan
- 2 Format for Final Project Reports
- 3 Editing Guidelines
- 4 Appendix 1. Methods for task analysis
- 5 Appendix 2. Getting to Know Users and Their Tasks
- 6 Appendix 3: Developing and Evaluating Initial Prototypes
Submit at Term Projects your project title (and short system name), list of team members (3-4, with email addresses and links to your wiki pages), and a paragraph or two describing your idea. Include a paragraph describing where or how you will get users who are appropriate usability test subjects. This project is a hands-on experience in iterative user-centered system design.
10%: User needs: Tasks, scenarios, and references (~4-8 pages)
Submit at Term Projects your results for this deliverable. Page estimate is just to set a rough expectation of how long this would be if printed. Grading is not based on length.
- Tasks. Describe in general terms the background to the interface. You should describe the expected users, their work contexts, and what they will use the envisaged interface for. Base this on Appendix 1.
- Scenarios. Describe 2-4 typical users doing typical complete uses of the interface. Based this on Appendix 2. Scenarios should cover a variety of the tasks (frequently done and important, infrequently done but still important).
- Provide annotated (1-3 sentences) references to a balanced set of the key 8-12 academic papers, industrial reports, and web pages on your topic. Compare your work to existing products. I expect you to become very familiar with the topic you choose, to know about commercial products and academic research work in refereed conferences and journals.
15%: First design (2-3 low-fidelity prototypes + description or tutorial ~6-12 pages)
From the task examples and requirements (following Appendix 3), your team should sketch out several competing interfaces. Discuss and choose the most promising of these, and develop a low-fidelity prototype (using storyboards) that demonstrates how the interface fulfills the requirements. Specifically, use the key users, their tasks, and the prioritized requirements as a type of requirements document to help you brainstorm prototypes that illustrate how your system would appear to the customer. You should be creating low-fidelity prototypes e.g., paper sketches, storyboards, cut-and-paste mockups. You should not be concentrating on prettiness or completeness; rather, you are trying to show the overall interaction style of your system. Each prototype should contain the core screens that illustrate how the system will work as a whole, including (perhaps) a sample interaction based upon some of the key tasks. Hint: To get diversity, each group member may want to try to create a few rough sketches before gathering as a group. You should also realize that some people may be better at this than others; this is not supposed to be a competition!
- Develop two very different low-fidelity prototypes that you believe will satisfy the major requirements. Discuss the prototypes with your team and (ideally) potential users.
- Your two low-fidelity prototypes can be hand-drawn or created using tools such as Powerpoint, Visual Studio, FrontPage, Flash, Paint, PhotoShop, etc. These should convey the visual design for your project and show the main screens (probably 2-8 screens for each design), with text labels, buttons, graphics, etc. They do not need to be funcational. An easy way to do this and explain the system is to write an introductory tutorial, which largely shows the main screens and explains what is happening in each one as the user carries out a common task sequence (such as logging in, doing a search, seeing the search results, and sending it to someone).
- Transition diagrams can give a good overview, be sure to provide at least one.
UPDATE 10/7: You should submit the original physical paper prototypes in class on the due date.
10% : Task list and questionnaires (~2-3 pages)
- Prepare the tasks for your usability test.
- Describe the scenario for your test subjects. Let them know what kind of user they will be pretending to be and what they need to do. Remind that that they are not being tested, it is the interface that is being tested.
- Give them a list of 3-8 tasks to perform. They may be simple sequence such as (1) login, (2) select a date and time for a medical appointment (3) send an email notice to yourself, (4) logout. They may be more complex, open-ended, and challenging tasks such as upload your audio recording of a book chapter, use the drawing tool to create an annotated diagram, or find patient histories that show no Papp smear tests within the last three years. Choose tasks that are clear enough so that users know when they are done and how they signal that they are done, for example clicking on a screen button, writing something down on your sheets, or simply saying that they are done so they can request the next task.
- Prepare the pre-test and post-test (submit the exact forms you will use).
- Pre-test: You may want a few pieces of personal data, such as gender, age, number of hours of internet use per week, number of years of professional experience as a physician. Your may also need to verify that they have proper vision, hearing, mobility, etc. This is so you can report about who your subjects were. Please refrain from personal identifying questions, that is, do NOT ask for name, social security number, address, etc.
- Post-test: You probably want to ask 3-8 questions, just so you get an idea of what reactions they had about the interface, what problems they found, and general comments. You can use semantically anchored 1-7 Likert scale questions to ascertain how difficult-easy they found specific aspects of the interface, or ask them to describe best and worst parts of the interface, or ask for general comments. Your goal is to get information that will help improve your interface design, and determine if it is acceptable or in need of further improvements.
20%: Usability test report (high-fidelity prototype + test ~6-12 pages)
- Create your high-fidelity prototype and submit screenshots plus your help screens that you used in the usability test. Run your usability test with 4-8 subjects (I'll be pleasantly impressed if you run more).
- Describe how the usability test went and any problems you ran into. For each subject give a one sentence description of who they were without any personally identifying information (e.g. 24-year old female college student, who bicycles to campus 2-4 times a week), followed by a half page description of how they did (e.g. where they had trouble) and what they said (e.g. "this is cool! I'd love to have this, especially if you added a print feature").
- Report on the post test questionnaire results using simple statistics and bar charts, plus extracts of key comments made by users. End by itemizing the problems you identified and prioritize them on a 5 point scale (5 is most important).
20%: Final report (revised prototype + description)
- Revise your prototype in response to what you learned from the usability study.
- Write a short description of what you revised, and specifically how those revisions respond to the problems you uncovered in the usability study.
The updated prototype should be linked to from the wiki, and you should post the description of changes on the wiki as well.
11/25 Update: per the announcement in class, the final report should be in pdf format.
11/29 Update: To reiterate as we discussed in class: This prototype and report is due on Wednesday 12/2 at 2pm.
11/29 Update: To clarify what this report should consist of, it should be a standalone report that can be read on its own. But it also can reference prior reports, and does not have to repeat everything in all of the prior reports. So, you should start by introducing the problem you are addressing, who your audience is, and what the problems are with competitive solutions. Then, summarize the problems you discovered in the usability study of your previous prototype and describe both your final prototype, and how you addressed the problems discovered in that usability study. Finally, describe open issues you are aware of, and what the next things you would do if you were to continue working on this project.
See schedule for due dates.
5%: Comments (Sign up to read and comment on another project, 1 page)
Your comment should be a constructive analysis of the report. Say what you liked about the project, and where you thought there was room for improvement. You can focus on any of the areas of the report (problem statement, usability study results, final product, or writeup).
Sign up at the Report Comments page.
15%: Presentation (Slide show presentation for class and web site)
- Title slide with name of teams members, emails, and date
- Statement of the problem & current solutions
- Approach to solution
- Demo (or slides of application)
- Usability test method
- Usability test results
- Conclusions & future work
Format for Final Project Reports
Revise your deliverables and prepare a comprehensive report:
- Title page: Title, Authors, Electronic Mail Addresses, and Date
- Abstract: 100-150 word overview of project
- Credits: Indicate who did what
Introduction (2-4 single spaced pages)
- Overview of the problem
- Discussion of previous work including (with references at the end):
- commercial systems
- previous academic papers
- relevant web sites
Presentation of design (8-20 pages)
- Give overview of your approach and solution.
- Show transition diagram for screens in your design.
- Present all screens you have designed with commentaries to explain (do screen grabs or scans of your prototype images).
- Present your tutorial/help
Report on development process (8-20 pages)
- Show a few of your low-fidelity prototype screens.
- Describe your process for arriving at the high fidelity prototype.
- Describe your usability testing process (include pre-test, task lists, post-tests).
- Describe each subject's experience and summarize the outcomes (list revisions).
Conclusions (1-4 pages)
- Describe the final status (what was implemented).
- Future work possibilities (what needs to be done to complete & refine).
- Recommendations to future developers of your idea.
Acknowledgments (a few sentences)
- Thanks to teachers, bosses, organizations, or friends who helped you.
References (5-20 references)
- In a neat standard format, alphabetical by last name of first author. Include peer-reviewed scientific journals, conference reports, trade press articles, white papers, websites, etc.
- Use standard wiki fonts, sections, and sub-sections.
- Number each figure and give it a caption.
- Cite references by authors and date, for example, (Jones and Berge, 2007) or (Microsoft, 2008).
Appendix 1. Methods for task analysis
Step 1. Generating a list of expected users, and an initial list of tasks. Interview knowledgeable people about their real-world tasks and observe them doing their tasks. Your goal is to generate an initial list of concrete task descriptions. You may or may not be able to access your real clients in the short time available for this exercise. Consequently, each team should select the approach below that best fits their constraints and team membership.
- The ideal: Interviewing the client. Get in touch with current or potential users. These users may now be using paper methods, competing systems, or antiquated systems for doing their tasks. Interview them about why and how they do their work activities, and what they expect out of a system. Ideally, this interview will occur while you are observing them do their work activities. These interviews and observations will give you some contact with customers and give you a feel for the real situation. This is more important than you think, for it will make you realize that "the user" is not an abstract notion, but real people with real needs and concerns. It will help you put a face on the faceless, and will help you understand where they are coming from.
- A reasonable alternative: Interviewing the client representative. When you cannot get in direct contact with end-users, you can use customer representatives instead. These will be people who have the most knowledge of the clients' needs. Examples are help desk people, or a worker's manager. However, it is crucial that the client representative has a deep and real (rather than idealized) understanding of what the workers actually do. People who work "in the trenches" with the staff are the best bet.
- When all else fails: Making your beliefs of the task space explicit. If you cannot get in touch with either real end-users or representatives, use your team members to articulate expected tasks. While this runs the risk of producing tasks that bear no relation to reality, at least you will get a diverse list of tasks out (because you have a diverse team), and it will put your beliefs and assumptions on the table. You can always show these to clients later, to see if these tasks indeed reflect what they do!
For whatever approach you chose, do the following steps. If you have a client and/or representative, you would do it with them. If you are "making it up", try to imagine as realistic a situation as possible.
- Have the client/representative/team recount a few (3-4) stories that typify the actual use of their system and/or process. Where possible, describe the people, the particular problems they wanted to solve, what information they brought into the meeting, the steps taken in the actual process, the constraints they had (e.g., time), what was produced, and whether they were satisfied with the outcome. All details are relevant. Alternatively, the task could be derived from direct observation of them doing their work.
- On a more general and less detailed level, list as many related tasks and their variations as possible.
- There will be many task variations in the list. Identify (with the user, if possible) which tasks are frequent, which are infrequent but still important, and which are rarer and not very important.
At this point, you will have a set of concrete, detailed examples of tasks that people now perform, or would want to perform on your system. Each task description should have the attributes described in the appendix and the second reading (see attached).
Step 2. Validating the tasks. The next step is to get a reality check of your task list. Have end-users and/or client representatives review your tasks. They should check to see if the set of people are representative of potential end-users of your product, if tasks capture the variations of those done by real people, and if details are realistic (they will be, if they are based on real customers!). You should ask for details that were left out of the original task description, get corrections, clarifications, and suggestions, and then re-write the task descriptions.
Note: This step is critical if you used a client representative or just yourselves instead of a real client. While it may not be possible for you to interview and observe many real clients, you can probably get one to comment on a compiled list of prototypical tasks.
Step 3. Deciding upon key users and a tentative list of requirements. The task examples will provide clues on specific system requirements that you could use to design your system as well as who your target users will be. Because it is unrealistic to meet all requirements and address all users, it is your job to prioritize them. From the task examples (and possibly by further discussion with end-users), decide upon the major system requirements and prioritize them into a) absolutely must include; b) should include; and c) could include. Similarly, decide upon what kind of users you must address, up to those you will exclude.
Good task examples:
- Says what the user wants to do but does not say how the user would do it
- you are not to make any assumptions about the system interface
- we will eventually use this to compare different interface design alternatives in a fair way
- Are very specific
- says exactly what the user wants to do
- we will eventually use this to specify the actual information the user would want to input to a system, and what information they will want out of it
- Describes a complete job
- should list all aspects of the task, from beginning to end
- this forces designer to consider how interface features will work together
- we will eventually use this to contrast how information input and output is carried through the dialog, i.e.:
- where does information come from?
- where does it go?
- what has to happen next?
- Says who the users are
- use particular people, if possible
- reflects real interests of real users
- the success of a design is strongly influenced by what users know and their real work context; we will eventually use this information to see if people realistically have the desire, knowledge and/or capabilities to accomplish their task with the system
Appendix 2. Getting to Know Users and Their Tasks
Example scenario for a clerk in a video store, including discussion. The eventual system will assist the clerks to perform their tasks.
Mary Farness, an experienced full-time clerk at the video store, opens the store in the morning. She begins the day by checking in all the videos returned in the night video slot, which typically number between 90 to 150 videos. She pauses her task whenever customers ask for her services. She usually checks in ten videos, and then reshelves them before going onto the next ten.
- Discussion. In this case, the "user" is the full time person who normally carries out this task. We expect them to be typical of an experienced clerk who will know the process well, and who will become well practiced at using the target system. The task is routine and frequently done.
George Marlay, a regular video store customer, approaches Mary and asks if they have the Frankenstein comedy video. She asks if he means "Young Frankenstein" by Mel Brooks, and he says yes. She then directs him to the shelf where the video is expected to be. George retrieves the video card and brings it to the front desk. Mary asks for George's membership card, but George has forgotten it. Mary then looks up his membership number. Mary checks out the video, but reminds George that he has not yet returned the video "Brazil", which is now a day late. George says that he will bring it in later today, and leaves with the video.
- Discussion. This task contains many typical clerk activities, which deals with vague requests about video titles, the location of the video in the store, forgotten membership cards, the video checkout activity, as well as reminders to customers about late videos. Most these tasks are frequently done, and important.
Anil, a part time clerk who works the telephone, comes in for an hour every third evening. His job is to search the rental records to find customers who are at least one day late on their video returns. For example, he phones Bob Jakobs, who is two days late. Bob answers, and Anil identifies himself, tells him that he still has the video "Volcano", and reminds him to return the video. Bob says he will bring it back in an hour or so, and Anil crosses his name off the list. He then phones Ania Sliven, and says (more or less) the same thing. However, Ania says that she has already returned the video the day before. Anil puts her on hold, runs to check the shelf and finds the video there. He apologizes and hangs up. He then phones Ang Lee, but there is no answer. He notes on another list that he should try this person again later. He continues in this manner. When he has finished the list, he starts again on those who have not answered.
- Discussion. This scenario identifies a specific activity that is less frequently done but still quite important. It also indicates that a non-regular staff member may be doing this task.
Why these are good examples:
- They say what the person wants to do, but does not say how it will be done. For later exercises, we can use these examples to see if a particular system would allow the person to accomplish their task.
- They are specific. They say exactly what type of information a person is bringing in to the task, exactly what information the person wants out of it, and how the person will use it.
- They describe a complete job. When we test a system, we can actually follow this sequence of events and see the amount of work that has to be done to get from one step to the next.
- They say who the user is. In this case, they identify particular people, their experience, and what knowledge/experience they have in their head. Again, this has implications to eventual use of a system.
Appendix 3: Developing and Evaluating Initial Prototypes
Step 1. Developing low-fidelity prototypes. Use the key users, tasks, and system requirements generated previously as a type of requirements document to help you brainstorm prototypes that illustrate how your system would appear to the customer. You should be creating several low-fidelity prototypes e.g., paper sketches, storyboards, or physical mockups. You should also be thinking about what conceptual model the system will portray to its customers. You should not be concentrating on prettiness or completeness; rather, you are trying to show the overall representation and interaction style of your system. Each prototype should contain the core screens that illustrate how the system will work as a whole, including (perhaps) a sample interaction based upon some of the key tasks. You should use the ideas of "psychology of everyday things" and "beyond screen design" to help you, as well as your general knowledge of other systems (there is nothing wrong with copying ideas, providing its legal!).
Hints. To get diversity, each of you may want to try to create a few rough sketches before gathering as a group. You may also want to talk to the systems people in your group, because there may be opportunities that already exist with the current way the information is stored and presented (e.g., if customers are already using software), or constraints that limit what you can do (e.g., if the system must be delivered as a Windows 95 application). You should also realize that some people may be better at this than others; this is not supposed to be a competition!
Step 2. Evaluate the prototypes. Your next step is to evaluate the prototypes.
- Discuss each prototype to see whether it is a possibility in principle (e.g., are there obvious problems with the conceptual model? Is it implementable?)
- Do a task-centered system walkthrough for each of your key tasks, and each of your user types.
- From the ones that are left, elicit reactions and further discussion from customers / counter people / appraisers. You may find that your end-users will tell you about further tasks and task details that were not thought about before!
Step 3. Reconsider priorities, and make a preliminary decision. Based on the prototype and evaluation exercise, you may wish to reconsider what customers you will address as well as what tasks and requirements you will support. It could be that you were wildly optimistic about what you could do! At this point, you should have a reasonable idea of which prototype styles are worth pursuing, or whether you should start again. Make your decision on what direction(s) to follow. If you have more than one direction, you may want to continue developing both a bit further. If you have no worthy candidates, return to step 2.
Hint. Don't feel committed to any prototype. This is the time where prototypes are quick to generate and cheap to discard. Use this time to explore the design space. While you may want to just get on with it, a bad design choice now can have disastrous and expensive repercussions later.
Step 4. Refinement and evaluation Refine your prototype by considering the nuances of each task, the functions that must be incorporated, and the expected reaction of each user. You may want to start considering the more subtle aspects of interface design at this point (e.g., psychology of everyday things, principles of graphical screen design, design principles). You should be continually evaluating these prototypes as appropriate. Real users should be commenting on them. You should be using walkthroughs, heuristic evaluation, and various observational methods.
If you follow this process, your prototypes will evolve from a competing series of design ideas, to a crude single design representation, to more realistic looking screen designs, to functioning systems. As your design is refined, you will move from very low-fidelity prototypes done on paper to medium-fidelity prototypes done online, likely through an interface builder. You will have detected and corrected the major interface design problems, and will be concentrating on fine-grained details. Eventually, you will create vertical prototypes with back-ends that fake some of the system functionality. To the user, however, it will look like the real system.