TwinList

From Cmsc734_11
Jump to: navigation, search

Logo.jpg

”It’s really impressive, the way you organize things is great. It’s definitely a step in the right direction ...”. - Med. reconciliation, A. Z. H

Team

Team Members:

  • Leonardo Max Batista Claudino
  • Sameh Khamis
  • Ran Liu
  • Ben London
  • Jay Pujara

Advisors:

  • Sigfried Gold
  • Catherine Plaisant

Resources

Project Report

Project presentation

Training Video

Demo

Source Code

Preliminaries

Proposal

TwinList: Multi-List Visualization and Reconciliation

In many applications, such as medication reconciliation, it is necessary to compare two or more lists and take some action on the list elements. When visualizing lists, some important constraints may include:

  • item attributes
  • relationships between items
  • uncertainty about item elements

In the example of medication reconciliation, attributes may include dosage and frequency, relationships may include drug interactions, and uncertainty may be the result of uncertainty in a patient's report. From the visualization interface, a user may wish to take some action – such as flag important items, add items to a master list, or remove items from the dataset.

We hope to create a prototype that addresses some of these issues related to list visualization. By starting with a simple solution that functions on item names and extending this version to include constraints mentioned above, our goal is to explore the various dimensions of this problem. Although we will use commercial tools such as Spotfire or Tableau to refine our ideas, we hope to implement a custom tool for this project. Our goal is to solve this problem generally, and evaluate our success using different datasets.

One dataset we hope to use involves medication lists supplied by Catherine Plaisant, who is involved in a medication reconciliation task through the SHARP project. We will solicit feedback from physicians to test hypotheses and evaluate the success of our approach. Another possible application of list visualization is showing salient trends in two disparate groups, such as words in a Twitter feed of Republicans versus Democrats.

Requirements Statement

Users should be able to do the following tasks:

  • Load data from a file
  • Look at the comparison of two lists of data
  • View the detail of a medication/word
  • Zoom in/out
  • Add medication to a action list
  • Remove medication from action list
  • Save action/export the action list
  • Change the argument value of the entries in action list

Discussion

Meeting Minutes

3/4/11

Discussion point: What did we learn from references?

  • Leo: Medical reconciliation, current research
  • Ben: Different strategies for Venn Diagrams
  • Jay: Disk Diagrams, value bars

Discussion point: What is our goal, what problem should we solve?

  • Just medical reconciliation?
  • General list comparison

Decided that general list comparison would align more with our interests and give more opportunities to explore.

Discussion point: What datasets can we use?

  • Prescription Lists - Catherine is still working on getting anonymized, real data
  • Twitter feed of Republicans & Democrats - mentioned in class
  • Gene data of expression levels in different tissues - from Professor Bravo, (Overview and Compare Tissues)

Discussion point: What aspects of the problem should we explore?

  • Dimensionality - long lists vs short lists
  • Number of lists - 2 or 3 is common, can we do more?
  • Similarity functions - binary, continuous, and multi-dimensional are all possible, how can we show them?

Discussion point: What visualization approaches should we evaluate for this problem?

  • Venn Diagrams
  • Similarity Matrices
  • Tag clouds and other spatial approaches to list placement, colorization

Discussion Point: Key functionality

  • Overview
    • Show "full" view of lists
  • Filter/Zoom
    • Show subset of list based on most/least similar items or other, attribute-based criteria (eg. medical reconciliation: dose frequency, drug category, warnings)
  • Details on Demand
    • Show most similar items to a selected item
    • Show all attributes of a given item

Discussion Point: What Toolkits can we use?

  • Ben is looking at the Java/Javascript InfoVis toolkit
  • Sam has used VTK in the past and recommends it for flexibility
  • Leo will see if PDE has an API

3/9/11

Discussion Point: Meeting scheduling

  • Agreed to the following meetings/milestones with Catherine (all at her office):
    • 3/18 @ 11 AM : Initial mockups of the three approaches we've proposed
    • 4/1 @ 11 AM : Prototype or Presentation Deck of functionality for first evaluation by domain experts (Collaborators in TX)
    • 4/8 @ 9:30 AM : Discuss feedback from domain experts

Discussion Point: Who does what by when

  • Leo: Look at similarity matrices and create rough mockup by 3/15
  • Ben: Look at Venn/Disk Diagrams and create rough mockup by 3/15
  • Jay: Look at Tag Clouds/List Placement and create rough mockup by 3/15
  • Ran: Take rough mockups from Leo, Ben & Jay and clean them up into final versions by 3/18 meeting
  • Sam: Research visualization toolkits and highlight key functionality/create examples by 3/18
  • Sam: Look at medication comparison algorithm Catherine provided and write a synopsis of important facets by 3/16

Discussion Point: Readings

  • Need to skim papers from huge reference list in Markowitz draft and see what's relevant
  • Leo/Ben/Jay should find more papers relevant to the particular approaches they're working on

3/17/11

Sam
Regarding the toolkit choice, the main support seems to be for Java and JavaScript, and recently for ActionScript/Flash because of a Prefuse port. VTK lacks most of the visualization that the other libraries have, and also has a big memory footprint because of all the scientific visualization (think bone segmentation, etc.) code that it has.



JavaScript seems to have 2 major players now, JIS and D3.JS. Even though D3.JS is new, it's moving quickly because it's written by the author of Protovis, another JS visualization library. Java has Prefuse and Piccolo, which also play nicely together. Social Action, for instance, uses both. ActionScript has its Prefuse port now, Flash anyone? :)

I like how clean the JS libraries are. Both support SVG and valid HTML, but you still have to run it in a browser, which makes me wonder about larger datasets. Prefuse Flare for ActionScript has a lot of support now; NameVoyager, ManyEyes, and BBC News all use it. My only issue with it is having to write a project in Flash!

And there is Java. Prefuse is here as well, and runs seamlessly in Swing windows. It also plays nicely with Piccolo, from Ben Bederson's group.

Ben

For what it's worth, I've used Flex (i.e. the framework for ActionScript/Flash) before and really liked it. It's a lot like JS in terms of syntax. If you are comfortable with Java/JS, ActionScript will be very familiar. The main problem with Flash is that it is closed-source and controlled by Adobe -- who are not getting along with Apple right now, if that means anything to us. (TwinList on iPad?) But problems aside, it is really easy to create beautiful interfaces with Flex.

3/18/11 Meeting with Catherine Plaisant

  • Have approximately 2 weeks to come up with something for evaluation by domain experts.
    • Could be a mockup/demo.
    • Should probably have a questionnaire in about a week.
  • Venn diagram:
    • Obvious shortcoming: sizing/placement; also, non-trivial to express intersection amount.
    • Possible workaround: use font color gradation/split to represent intersection amount.
    • Should create a mockup using medical data (2 lists).
  • List Placement
    • Value bars: perhaps confusing at first, but a domain expert would pick it up quickly.
    • Is it clear what is exactly the same? Should jump out at you.
    • Solution: put exact matches in the center instead of duplicating.
    • Suggestion: highlight only things that are different.
    • Dosage could be only difference.
    • Should extract dosages from provided list, for mockups.
    • Regardless of length of list, our visualization should show only the top n items (set in preferences or interface)
    • Can compare medical reconciliation to text similarity by comparing:
      • dosage <=> term frequency
      • category <=> part of speech
    • 2 levels of equivalence: exact, approximate
    • How do we represent matches between >2 items?
      • In List Placement model, should bring items closer together.
      • Easier to visualize in similarity matrix.
  • Similarity Matrix
    • Pivot/sentinel list allows for comparison of 3 lists.
    • Suggestion: sentinel list should be color-coded to reflect similarity.
    • Should have ability to re-order/sort/"roll/dial" the sentinel list.
    • Not entirely clear which attribute(s) of items are different.
      • Could be a details-on-demand solution: click/mouse-over gives details of similarity between items.
      • Tension between simplicity and detail.
      • Solution 1: Subdivide each cell and use hue/gray-scale or even just actual numerical values to indicate amount of similarity.
      • Solution 2: Provide drop-down list to filter which attribute is being used for similarity.
  • Grouping:
    • What about items that belong to multiple groups?
  • Decision-making:
    • 3 states for each item: in/out/undecided.
    • Preference for automated behavior on initialization.
    • Mouse-over should display options to set approval state.
    • Note: approval state should be mirrored in relative positions.
    • Once removed, item should be grayed-out (but still visible).
  • Moving forward:
    • Should rally behind one design.
    • Catherine is very fond of spatially-sensitive word cloud (refer to Drew Conway solution).
    • Come up with more detailed mockup, illustrating scenarios/use-cases.
    • Deliver detailed mockups by April 1, to collect feedback from physicians.
  • Work breakdown:
    • Jay/Leo: backend framework.
    • Ran: story-board, use-case illustrations.
    • Sam/Ben: front-end exploration. Download libraries and play around with them -- Sam in JS; Ben in Flex. Reconvene to discuss pros/cons of each.
    • Jay: setup repository.
  • Programming language:
    • Need to decide between Java/JS/Flex.

4/25/11

Comments from Catherine

  • Reconciliation animation
    • Control speed
    • Add buttons to walk through the stages
    • Change button labels to "Find matches" or "Match lists" and "Reset view"
  • Accept/Reject tweaks
    • Add icons or use strikeout
    • Grayed-out should be "grayer" to be easier to distinguish from normal
    • Accepted/Rejected should be at the top of the Action List
    • Animate acceptance?
  • Viewport options
    • Fit to text
    • Fix header overlaps
    • Horizontal scroll of lists
    • Pad whitespace to deal with large right column, accept reject off screen
    • Text on 1/2 lines
    • Color tweaks?
    • Check functionality at 1024x768
  • Group by

Evaluation

  • Two medical professionals - friend who does many reconciliations and researcher

Potential Comparison Measures

Log-Likelihood Ratio

The log-likelihood ratio was used to measure the significance of the difference in word frequencies between an analysis text and a reference text in WordHoard. This measure is symmetric, which means that switching the two texts will not change the value. WordHoard used the log-likelihood ratio, scaled, to determine the word size in a tag cloud. Color was also used to differentiate between words more common to one text than the other. In comparing list items, we can replace the word frequencies by a certain attribute value to calculate a similar measure.

The log-likelihood ratio denotes the significance of a difference. Other test statistics or score function, like Pearson's Chi-Square test or the F-Test, can also be used. However, if the significance of the difference is not motivated, and instead the visualization of the difference itself is sought, standard distance metrics, like the absolute difference or the squared difference, should be used instead.

Potential Visualization Paradigms

Dimensionality Similarity Multiple Lists
Venn Diagrams Multidimensional data can be represented by the color spectrum: one color per dimension. (Continuous) Degree of similarity can be represented by font size: larger font sizes would indicate stronger similarity. Work for up to 4 lists; beyond that, it is impossible to visualize all intersections in one diagram. Multiple pairwise diagrams could overcome this limitation, but only up to a point; beyond a certain number, screen space will be exhausted and it will become difficult for the user to comprehend relationships between multiple pairwise combinations.
Lists Techniques such as Value Bars can be used to show multidimensional data. Coloring and sizing items can also provide attribute information. Long lists cannot be displayed at once, but filters and grouping/ordering can help users process larger lists. Item placement can indicate unique items, or similarity between items, with alignment showing equality and distance representing difference. Showing similarity between all pairs of items is generally not possible. Multiple lists can be show in parallel, but list placement and value bars become difficult or impossible to use. The degenerate case is simply using alignment to show duplicates among many lists, with indicator icons for attribute differences. Multiple pairwise comparisons are possible.
Similarity Matrices Dealing with multi-dimensional items would be somewhat challenging for this visualization, if not prohibitive, since the elements of the similarity matrices are scalars by definition. However, multi-dimensionality could be supported at least within the sentinel list, by applying strategies such as treemaps. Similarity scores should be real non-negative numbers and should be represented by some color-coding. With a sentinel list A and 2 similarity matrices, one can visualize 3 lists at the same time. Matrices will represent the similarities AxB and AxC. The display would show a couple of items at once. To cope with long lists, the user should be allowed to scroll the list items or similarities by vertical panning (that would scroll items in the sentinel list and on both matrix simultaneously) and horizontally (that would scroll an individual similarity matrix). Furthermore, items should be alphabetically ordered.

Whenever the user selects an item either from A or a similarity score, the system should spot corresponding items across lists.

Potential Tools/Libraries

The following is a list of potential visualization toolkits/libraries that could be used for this project. This will help determine the framework we use. Combining multiple toolkits is also possible. An example would be the Social Action visualization that used both Prefuse and Piccolo.

Javascript

  • JavaScript InfoVis Toolkit This seems to be the most popular visualization toolkit for JS. Creating custom visualizations requires some hacking.
  • Protovis Its major drawback is using custom tags, which makes debugging the library output harder.
  • d3.js Nice scatter plot matrix in this one. D3.JS is also written by the author of Protovis and is aimed to supersede it. Debugging outputs is easier.
  • Processing.js More general drawing library, and can be used for visualization. It doesn't contain, however, the standard visualizations other packages might contain.
  • Standard JS libraries like Dojo and jQuery also have visualization plugins.

Java

ActionScript/Flash

  • Prefuse Flare Used by BBC News, Wired, Many-Eyes, and NameVoyager for some visualizations.

Python

  • VTK is a C++ open source class library for visualization and computer graphics. Python bindings are officially supported. VTK also can be displayed in WxPython windows. VTK is mainly a scientific visualization toolkit, and therefore its support for information visualization is still immature.

Project Stages

Mockups and Prototypes

Venn Diagram

TwinList Venn Diagram Mockup.png

[PDF Version]

The above image illustrates how venn diagrams may be used to visualize the intersections of multiple textual sets (lists). Here we see a mockup of one using rhetoric from political parties. Each term is a word/phrase that occurs frequently in their respective rhetoric. The size of the font indicates the overall frequency of the term in the corpus. (Larger font implies higher frequency.) The size of the intersections reflects the total number of common occurrences in the respective sets. Thus, in this image, we find that the most common term is "jobs", which is common to all three parties. Another common term is "healthcare", though it is not mentioned as much as "jobs". We also see that there is not much overlap between all three parties; in fact, there is no overlap at all between the rhetoric of the Democrats and the Tea Party; meanwhile, there is a large overlap between the rhetoric of the Republicans and the Tea Party. In the disjunctions, we see terms that are exclusive to each party, such as "green jobs" and "education" with the Democrats and "birth certificate" and "kenyan" with the Tea Party.

Overview:

  • Size of font indicates overall frequency of term in corpus: term_freq(word) = \sum_{set \in Sets} term_count(word, set)
  • Size of intersection illustrates size of intersection of terms and counts: intersection(set_A, set_B) = \sum_{word \in Lex} common_occurrences(word, set_A, set_B)
  • Different color for each set. (Works great with 3, since there are 3 primary colors.)
  • Size of circle should indicate size of set, but there are issues with this. (See cons below.)

Pros:

  • Intuitive, well-known visualization

Cons:

  • Does not extend to more than 4 sets; in fact, even 3 sets are difficult to visualize.
  • Sizing is definitely non-trivial:
    • Sizes of circles/intersecting regions should indicate sizes of sets/intersections, but it becomes difficult to maintain a linear ratio due to the text.
    • For instance, if a word there is very little overlap between two sets, but one of the terms in the intersection is a high frequency term (i.e. large font), how do we size the overlap? Even if the term is only mentioned once (i.e. small font), but it is a very long word, it still takes up a lot of space.
    • One workaround might be to let it spill out of the container, but this looks sloppy.
    • Another workaround is to use heat-mapping (i.e. color coding) to indicate frequency of term, but then color cannot be used to represent multiple variable matching.
  • Intersection does not illustrate the frequency of term in individual groups.
    • For instance, if "jobs" is used more by Republicans than Democrats, this is not evident by this image.
    • One workaround is to use the placement of the term from the center of the intersection as an indicator of frequency imbalance. However, this leads to yet more sizing/placement issues.

List Placement

ListPlacementMockup.png

The List Placement visualization uses placement of items to show item alignment. Items are aligned if they are judged equal by a similarity algorithm. We could also use slight offsets to show "fuzzy" equality, as between a brand name and generic drug, or words with different stems. The distance of an item from the center of the visualization indicates the "uniqueness" of the item, allowing a user to quickly pick out which items are unique to a list, or are present in both lists but have different attributes. A possible issue is that the variable sizes of items may make this difficult to implement. Another possible issue is that a large number of items may make the visualization hard to use. Since the distance from the "center" is used to indicate uniqueness, this method would not work well for more than two lists.

Multi-attribute data is presented using Value Bars, with continuous variables placed in discrete buckets. The Value Bars can use highlighting to make disagreeing attributes stand out. One issue is that for data with many categories, Value Bars would need many colors. Having a legend for each color for each Value Bar might require too much space, while using tooltips might make a user's workflow too slow, and it's easy to imagine novice users being inundated by the Value Bars. Another issue is that if an item can have multiple values for the same attribute (eg. side effects of a drug), then the Value Bar may become crowded and illegible. The Value Bar presupposes that we have attribute information for each item, but we may need an indicator to show missing data.

A filter panel allows the user to view a subset of the list based on the pertinent criteria. Items can be grouped, ordered and sized based on different dimensions or metrics - such as dosage or "uniqueness". These can help deal with issues associated with large lists, but focusing on a particular set of drugs before moving on. A details window allows granular information about a specific list item to be displayed. Alert icons could be used to flag exceptional conditions, such as a risky drug interaction or dosage beyond recommended range.

Similarity matrices

Mockup-sim-mat2.jpg

[PPT file]

This simple visualization combines a list and 2 similarity matrices. The display comprises a sentinel list (A) in between two similarity matrices (AxB, AxC), where B and C are lists. Elements in AxB, AxC reflect the similarity between one of its items an item in the sentinel list. According to the color code presented, the more red is the element in the similarity matrix, the closer it is to the row element in A. The whiter, the farther. Note that if an item in B that matches another in A has the same color to a corresponding item in C, that means the 2 elements in B and C are the same. In other words, based on how B and C relate to A, the user can directly infer the relationship between B and C.

In user selection mode, an item can be selected in the sentinel list by clicking on the corresponding element. The corresponding row on both matrices will stand out by making others transparent. The best matching items will be highlighted in the headers of B and C and linked to the corresponding elements by a line coded with the similarity.

List Placement Visualization v.2

Visual.png
  • Instead of repeat the same entries of the list, we only show one name in the middle. (This is for completely-the-same entries).
  • Only show the difference for the partial-similar items.
  • The most far apart items are unique items which only appear on one list. Show both the names and arguments values.

List Placement Visualization v.3

Sig of both similar items displayed
Sig of left item plus the difference


  • Lists are broken down to sets "Same" (green), "Similar" (pink) and "Unique" (red).
  • Show “same” items only once.
  • Highlight differences in the sig.
  • Note the Calcium Carbonate entries on the figures above: sigs of both items can be shown (top-left figure) or, alternatively, the sig of the left item can be displayed together with the dosage difference (top-right).

Design Sketches

Version 1

Interface v1.png

This is the first interface mockup. In the visualization window, we used list and valuebar as our example. The left-hand side of the visualization is the home list (patient list), and the right-hand side is the doctor list. User can select a single item of the list, and the detail of that medication will be shown on the detail panel on the top right corner. User can add the selected item to the reconciled list by both click on the add button to the right, and drag-and-drop the item in the table below the visualization window. User also can remove a item from the reconciled list by click on the remove button. The bottom right panel is a group of filters allow user to filter the data.

Version 2

Interface v2.png

After analyzing the first interface, the group agreed that it is difficult to visualize uniqueness in a continuous manner. The new interface version attempts to address the problem by breaking the lists down to three sets, colored accordingly: same (green), similar (pink) and unique (red). "Same" items are shown only once. Differences are highlighted in the sig. For similar pairs of items, either both sigs will be shown or the first plus the difference between the two. This is yet to be decided. In terms of reconciliation, users will be able to interact with the list item-by-item. By right-clicking an element, they will be able to Accept, reject or mark for reconsideration. We are also considering default behaviors such as automatic acceptance of "same" or "unique" items.

UI Storyboard

Load data

  • Task description: User can load data by select files and upload the file to the software.
  • Precondition: User need to be authenticated.
  • Sub-tasks:

1. click on "load file" button/Menu>load file

2. enter the number of list user want to load

Loaddata1.png

3. choose file to load.

Loaddata2.png

4. look at the preview to make sure the data format is okay

5. click on "load data" to load data

Visualize

  • Task: show visualization of the lists.
  • Precondition: User must loaded data
  • Sub-tasks:

1. go to the visualization page. (maybe automated show after user load data successfully)

Visual.png

view details

  • Task: view the details of one item(all the argument values and other information)
  • Precondition: the visualization is shown
  • Sub-tasks:

1. selected the target item in the visualization

Vis selected.png

2. the details of the selected item will be shown in the details panel.

Detail.png

3. user can add the selected item to the action list by click on the "add to action list" button.

Detail selected.png

Change the order/position of items in the visualization

  • Task: change the order/position of the list shown in the visualization
  • precondition: the visualization is shown and the control panel is shown
  • Sub-tasks:

1. go to the control panel and click on "sort by" button.

Group.pngGroup selected.png

2. choose the preferred option from the drop down menu. eg. sort by similarity.

Vis by sim.png

3. the visualization will change according to user's choice.

Sort by sim.png

Add item to action list

  • Task: User can add items to an action list
  • Precondition: the visualization is shown
  • Sub-task:

1. select an item: user can select the item by: double click on the item drag-and-drop the item to the action list panel right-click on the item and select "add"

2. the selected item will be add to action list. Actionlist.png

3 quick add. In order to allow user to make decision fast. Every time the user hover on the item in the visualization, a floating panel will show. User can click on check mark to add this medication to the action list, click on the cross to remove this item from the action list, or click on "?" to deal with it later.

Quick decision.png

Interacting with visualization list and reconciliation table

The following screen demo illustrate how the system is currently being planned to allow item reconciliation.

1. The user right click one item in the list. He/she is prompted to accept, reject or mark it to reconsider.
Reconciliation-list-1.png

2. Suppose the user chooses to accept (Note "Accept" highlighted).

Reconciliation-list-2.png

3. The accepted item is then sent to the reconciliation table (blue table on the bottom) and is made transparent in the visualization list.

Reconciliation-list-3.png

4. Another option is to automatically accept "Same" and/or "Unique" items automatically.

Reconciliation-list-4.png

Evaluation Methods

Preliminary evaluation: survey

After our last meeting with Dr. Plaisant, we have decided to work on a questionnaire that would allow her to collect feedback from some of the doctors that collaborate with the project. Since they are potential users, their opinion is crucial. The selected questions involve interface design and usability issues that will guide our initial development:

1. Should we include the value bars on the sides in the final interface, or should we discard them and use tooltip text?

2. Should we show the attributes that are different for the similar items next to the item name in the main window?

3. Should the attributes that are different be shown in a tooltip text only for each similar item?

4. Should the size of list items stay fixed for easier correspondence, or should the size signify the severity of the difference or any other attribute?

5. Should we automatically add the same and the unique items to the reconciled list?

6. Should accepting one similar item discard the corresponding one from the other list, or should they be handled separately?

7. Is having three distinct sets for same, similar, and unique items easier to visualize than measuring the uniqueness by the distance from the center?

8. Is it preferable to reconcile multiple lists a pair at a time, or all side by side?

9. Are there any other features you would like to see?

10. Are there any features you find distracting or less useful?

Evaluation Results

References


  • References to Clean Up
[1] Roughead EE, Semple SJ. Medication safety in
acute care in Australia: where are we now? Part 1: a
review of the extend and causes of medication
problems. Aust New Zealand Health Policy.
2009;6:18.
[2] De Koning JS, Klazinga NS, Koudstaal PJ, Prins
A, Dippel DW, Heerings J, et al. Quality of care in
stroke prevention: results of an audit study among
general practitioners. Prev Med. 2004;38:129-36.
[3] Poon EG. Medication reconciliation: whose job is
it? AORN J. 2009 Jun;89(6):1180, 1122.
[4] Ellitt GR, Engblom E, Aslani P, Westerlund T,
Chen TF. Drug related problems after discharge
from an Australian teaching hospital. Pharm World
Sci. 2010.
[5] Joint Commission on Accreditation of Healthcare
Oganizations’. Using medication reconciliation to
prevent errors. Sentinel Event Alert. January 25,
2006;(35):1-4.
[6] Ponovost P, Weast B, Schwarz M, Wyskiel RM,
Prow D, Mianovich SN, Berenholtz S, Dorman R,
Lipsett P. Medication reconciliation: a practical tool
to reduce the risk of medication errors. 2003;
18(4):201-205.
[7] Royal S, Smeaton L, Avery AJ, Hurwitz B,
Sheikh A. Interventions in primary care to reduce
medication related adverse events and hospital
admissions: a systematic review and meta-analysis.
Qual Saf Health Care. 2006;15:23-31.
[8] Keohane CA, Bates DW. Medication safety.
Obstet Gynecol Clin North Am. 2008 Mar;35(1):37-
52.
[9] Gandhi TK, Weingart SN, Borus J et al. Adverse
drug events in ambulatory care. N Engl J Med. 2003
Apr 17;348(16):1556-64.
[10] Peyton L, Ramser K, Hamann G, Patel D, Kuhl
D, Sprabery L, Steinhauer B. Evaluation of
medication reconciliation in an ambulatory setting
before and after pharmacist intervention. J Am Pharm
Assoc. 2010 Jul-Aug;50(4):490-5.
[11] Steurbaut S, Leemans L, Leysen T, De Baere E,
Cornu P, Mets T, Dupont AG. Medication history
reconciliation by clinical pharmacists in elderly
inpatients admitted from home or a nursing home.
Ann Pharmacother. 2010 Oct;44(10):1596-603.
[12] Bailey AL, Moe G, Moe J, Oland R.
Implementation and evaluation of a communitybased
medication reconciliation (CMR) system at the
hospital-community interface of care. Healthc Q.
2009 Oct; 13 Spec No:91-7.
[13] Kripalani S, LeFevre F, Phillips CO, Williams
MV, Basaviah P, Baker DW. Deficits in
communication and information transfer between
hospital-based and primary care physicians:
implications for patient safety and continuity of care.
JAMA. 2007 Feb;297(8):831-41.
[14] Bayoumi I, Howard M, Holbrook AM, Schabort
I. Interventions to improve medication reconciliation
in primary care. Ann Pharmacother. 2009
Oct;43(10):1667-75.
[15] Johnson CM, Marcy TR, Harrison DL, Young
RE, Stevens EL, Shadid J. Medication reconciliation
in a community pharmacy setting. J Am Pharm
Assoc. 2010 Jul-Aug;50(4):523-6.
[16] Pronovost P, Weast B, Schwarz M, Wyskiel
RM, Prow D, Milanovich SN, Berenholtz S, Dorman
T, Lipsett P. Medication reconciliation: a practical
tool to reduce the risk of medication errors. J Crit
Care. 2003 Dec; 18(4):201-5.
[17] Weeks DL, Corbett CF, Stream G. Beliefs of
ambulatory care physicians about accuracy of patient
medication records and technology-enhanced
solutions to improve accuracy. Healthc Qual. 2010
Sep-Oct;32(5):12-21.
[18]Bates K, Beddy D, Whirisky C, Murphy M,
O’Mahony JB, Mealy K. Determining the frequency
of prescription errors in an Irish hospital. Ir J Med
Sci. 2010 Jun;179(2):183-6.
[19] Ansari M, Neupane D. Study on determination
of errors in prescription writing: a semi-electronic
perspective. Kathmandu Univ Med J (KUMJ). 2009
Jul-Sep;7(27):238-41.
[20] Rosa MB, Perini E, Anacleto TA, Neiva HM,
Bogutchi T. Errors in hospital prescriptions of highalert
medications. Rev Saude Publica. 2009
Jun;43(3):490-8.
[21] Calligaris L, Panzera A, Arnoldo L, Londero C,
Quattrin R, Troncon MG, Brusaferro S. Errors and
omissions in hospital prescriptions: a survey of
prescription writing in a hospital. BMC Clin
Pharmacol. 2009 May 13;9:9.
[22] Shneiderman B, Plaisant C. Designing the User
Interface. 4th ed. Boston, MA : Pearson Addison-
Wesley; 2005.
[23] Rubin J. Handbook of uability testing. New
York: John Wiley & Sons, 1994.
[24]Sauer J, Selbel K, Ruttinger B. The influence of
user expertise and prototype fidelity in usability tests.
Applied Ergonomics. 2010;41(1):130-140.
[25]Catani MB, Biers DW. Usability evaluation and
prototype fidelity: users and usability professionals.
Proceedings of the Human Factors and Ergonomics
Society 42nd Annual Meeting; 1998 Oct 5-9; Chicago,
IL. 1331-1336.
[26] Virzi RA, Karis D, Sokolov JL. Usability
problem identification using both low and highfidelity
prototypes. Proceedings of CHI’96; 1996;
Vancouver, BC, Can. 236-243.
[27] Walker M, Takayama L, Landay JA. Highfidelity
or low-fidelity, paper or computer? Choosing
attributes when testing web prototypes. Proceedings
of the Human Factors and Ergonomics Society 46th
Annual Meeting; 661-665, 2002.
[28] John BE, Kieras DE. Using GOMS for user
interface design and evaluation: which technique?
ACM Transactions of Computer-Human Interaction
(TOCHI). 1996;3(4):287-319.
[29] Card SK, Moran TP, Newell A. The keystrokelevel
model for user performance time with
interactive systems. Communication of the ACM.
1980;23(7):396-410.
[30] Kieras D. Using the keystroke-level model to
estimate execution times. Unpublished report. Ann
Arbor: Department of Psychology, University of
Michigan; 2001.
[31] Legacy Health. Collateral Materials for
Medication Reconciliation.
http://www.legacyhealth.org/body.cfm?id=757.
Updated January 4th, 2010. Accessed December 9th,
2010.
[32]Poon EG, Blumenfeld B, Hamann C, Turchin A,
Graydon-Baker E, McCarthy PC, et al. Design and
implementation of an application and associated
services to support interdisciplinary medication
reconciliation efforts at an integrated healthcare
delivery network. J Am Med Inform Assoc. 2006
Nov-Dec;13(6):581-92.

Deliverables

Credits

All members of the team were actively involved in each stage of our project. In particular, each member contributed in some way to the exploratory visualization design, client interaction, software implementation, user testing, write-up and general group organization.