From DeviceLab
Jump to: navigation, search

Project Overview

In this age of computers, paper is still extensively used by people for many tasks such as note-taking, document reviewing, prototyping, and social contact. The reason behind this paradox is the gap between the paper and computer affordances: each of them has unique merits that the other can not provide.

In response,PapierCraft aims to bridge paper and digital world in a natural and seamless way, so that people can enjoy the advantages of the two media, and information can flow between them without inefficient transcribing. The key point is "Paper-based Interface". We take a paper document as a "proxy" of its corresponding digital copy, so that users' handwritings on paper are transparently captured and merged back to the digital version. Along this line of research, we further propose a paper-based pen gesture command system well adapted for paper, which allows users to issue digital commands like copy/paste, hyperlinking, stiching, email and even googling directly on paper. The envision is to treat paper as a mobile physical computing device, which can interact with other computers, PDAs, Cellphones and so on, building a information space spanning the physical and digital world.

To achieve this goal, we are exploring the following research problems:

  • A mobile infrastructure supporting digital interactions on paper without constrains on user's existing pen/paper practices
  • A command system which is naturally integrated with pen/paper interactions and, especially, works with weak real time feedback of paper
  • A highly portable multi-modal feedback mechanism to address insufficient dynamic visual feedback on paper
  • A framework to support mobile users migrating among different working environments with diverse computing resources
  • A longitudinal user study on the paper interface in people's real life


command system multi-modal feedback pen link paper and mobile devices replace a tablet PC with paper interface?

Group Members

  • Brynn Evans, Graduate Student, Dcog-HCI Lab, Dept. of Cognitive Science, UCSD
  • Kevin Li, Graduate Student, Dept. of Computer Science, UCSD
  • Chunyuan Liao, Graduate Student,HCIL & Dept of Computer Science, UMD
  • Carsten Röcker, Post-Doc, Dcog-HCI Lab, Dept. of Cognitive Science, UCSD
  • Jill, Undergraduate Student,Dcog-HCI Lab, Dept. of cognitive Science, UCSD
  • Tim, Graduate Student,Dept. of Computer Science, UCSD


  • Microsoft Research (as part of the Microsoft Center for Interaction Design and Visualization at the University of Maryland)
  • NSF under Grant IIS-0414699 and IIS-00447730

Work in Progress

  • Chunyuan:
    • Refine PapierCraft 2.0 for longitudinal user study
    • Building realtime feedback control board for Logitech Bluetooth pen
  • Francois: being supervisor
  • Brynn:
    • Test PapierCraft 2.0
    • Locate participants for longitudinal user study
  • Kevin:
    • Write a named Bluetooth serial port service on iPAQ, to which a Logitech/Nokia pen will connect
  • Casten:
  • Jim: being supervisor


  1. PaperCP: Exploring the Integration of Physical and Digital Affordances for Active Learning,Chunyuan Liao, François Guimbretière, Richard Anderson, Natalie Linnell, Craig Prince and Valentin Razmov, Proceedings of INTERACT’07, in press. paper
  2. PapierCraft: A Gesture-Based Command System for Interactive Paper,Chunyuan Liao, François Guimbretière, Ken Hinckley and Jim Hollan, (Accepted by ToCHI in June 2007)paper
  3. Pen-top feedback for paper-based interfaces,Chunyuan Liao François Guimbretière and Corinna Loeckenhoff,Proceedings of UIST’06, pp. 291 - 230.paper video
  4. ButterflyNet: A Mobile Capture and Access System for Field Biology Research,Ron B. Yeh, Chunyuan Liao, Scott Klemmer, François Guimbretière, Brian Lee, Boyko Kakaradov, Jeannie Stamberger, and Andreas Paepcke. Proceedings of CHI’06, pp. 571 – 580. paper
  5. PapierCraft: A Command System for Interactive Paper,Chunyuan Liao, François Guimbretière and Ken Hinckley. Proceedings of UIST’05, pp. 241 – 244. papervideo
  6. ProofRite: A Paper-Augmented Word Processor,Kevin Conroy, Dave Levin and François Guimbretière. unpublished papervideo
  7. Paper Augmented Digital Documents,François Guimbretière,Proceedings of UIST 2003, pp. 51 – 60.paper slides

Using PapierCraft

What you need before flying with PapierCraft

  1. A Linux server. PapierCraft server has been tested on Redhat 2.4.21
  2. A Windows PC running Windows XP (SP2). Vista has NOT been tested
  3. A B&W or Color laser printer for printing Anoto pattern paper
  4. A COLOR INKJET printer whose black cartridge has been removed and “black” is simulated by Cyan/Magenta/Yellow. This printer is used to print calibration pages and PDF documents on the above pattern paper.

For user interactions, you need:

  1. One digital pen. Nokia and Logitech IO digital pens have been tested in our prototype
  2. Several pieces of unused Anoto pattern paper
  3. The *.PAD ( Page Address Definition ) file associated with the above pattern paper.

If you don’t know how to get the pattern paper and PAD file, please refer to Get Started[Obtain Anoto pattern paper].

What is in the PapierCraft software suite

The basic components of the runtime system include

  • PADD server: store the printed digital documents,annotations and physical-digital page mapping information
  • Acrobat Plug-in, capturing printing information and rendering proofreading marks retrieved from PADD server
  • A virtual printer, which collaborates with the Acrobat Plug-in to capture printing information.
  • Digital pen synchronization facility, which works with the Nokia/Logitech digital pen software suite to download pen strokes from the pen and submit them to the PADD server(see below)

Below are components for PapierCraft 2.0 and later

  • Event server: collaborates with the synchronization facility, and coordinates pen commands on different paper documents and/or computers for across-document operations, like copy/paste and hyperlinking
  • SnippetViewer: users can review, organize and search snippets created on paper by drawing annotations and tag commands
  • SessionViewer: users can play back and review pen writings for a session ( period between two synchronizations of a pen), and fix gesture recognition error if any

<to do: architecture diagram>

Get Started

  1. Obtain Anoto pattern paper
  2. Compile and Install Software at Server Side
  3. Install Prerequisite Software for Client Side
    1. Nokia or Logitech Pen software suite, depending on what pen you have. NOTE: ONE and ONLY ONE kind of pen software on each computer, since the Anoto SDK does not support more than one brand on a single PC.
    2. Anoto SDK 3.1 or later. NOTE: it MUST be installed after the above pen software
    3. Acrobat full version 6.0 or later
    4. Adobe Acrobat Acrobat SDK 5.0 or later
    5. (Ver 2.0 and later only)Windows XP Tablet PC Edition SDK 1.7
    6. (Ver 2.0 and later only)Microsoft Windows XP Tablet PC Editor 2005 Recognizer Pack
    7. (Ver 3.0 and later only)Microsoft Speech-API 5.1 and later
  4. Install/Compile PapierCraft software suites below

Known Bugs and Issues

  • "SDK Test License locked" during pen synchronization

Such message may be reported when you synchronize the digital pen. It is safe to press "OK" to simply ignore it.

  • "Transformation matrix not found"

It happens when the PADD printing facility can't find the precise "physical-digital" coordinate transform matrix. You can always click "OK" to use the default one, which is usually good enough

  • "Document failed to upload"

NOTE: the PDF file name MUST be shorter than 30 characters due to a bug in the server program, otherwise you will get this problem. If this is not your case, please check the availability of the servers.[This bug has been fixed !!]


  • Q: How can I reset my Logitech digital pen?

A:Follow the steps below

  1. press the pen tip against a hard surface like a table and hold
  2. press and release pen's "reset" button ( above the pen's 4-pin plugin interface )
  3. the pen's 2 LEDs will flash in red, just keep holding it until the LEDs change to red and green alternately
  4. put on pen cap and wait for several seconds, and the LEDs will be off
  5. the pen is ready.

  • Q: How can I tell if my Logitech IO2 pen support Bluetooth?

A: Two ways: First the Bluetooth pen body is in dark blue, but non-bluetooth one in dark grey. See their pictures: Bluetooth pen and Non-bluetooth pen

Secondly, when you take off the pen cap, the Bluetooth pen will flash a built-in blue LED above the Logitech logo,but non-bluetooth pen doesn't