Object Service Architecture
Web Annotation Service
Venu Vasudevan and
Object Services and Consulting,
September 15, 1998
Military intelligence analysis and command and control
organizations as well as industrial and educational institutions increasingly
use the World Wide Web to share and make available information about their
"organizational memory" or the situations they monitor. Today's Web
is a global document repository where authors actively publish information
and browsers passively view it. The objective of the OSA/Annotations
project is to augment the Web with the basic capability for third party
viewers to augment or annotate content supplied by authors, with or without
author knowledge. This can allow closed groups of collaborators or
large "communities of interest" to actively add expertise to information
posted on the Web so that the community information base can rapidly and
incrementally improve based on the expertise of all its members, not just
primary authors. Our approach is to develop an open Annotation service
that can be used in conjunction with standard Web browsers. The service
is invoked by a general purpose URL interceptor which
installs the side-effect (transducer or extension) of searching for document
annotations as the document itself is retrieved. The basic annotation
service can be tailored in a number of ways, for instance, to only display
annotations from certain groups or of certain types. Higher-level
facilities like pedigree drill down and filtering, argumentation, or workflow
can be built using the OSA/annotations service as a primitive.
In the JTF/ATD Web Server project,
DARPA is using an idiosyncratic object-oriented "web" to represent and
monitor situation description and assessment information about evolving
command and control events. Though it is much less structured at
present, information authors use the World Wide Web for much the same purpose,
to post information content about topics of interest within private Intranets
or to the public Internet. To date, Webs are populated by authors
who are authorized to add content. Surfers who access the information
provided on Web pages are generally passive viewers, with no direct way
to add their expertise to the subject matter. Yet these same viewers
have an interest in the subject matter they are accessing and may have
additional knowledge (more recent, deeper, better analyzed, corrections,
viewpoints, clarifications, ratings, filters) that could be shared across
a community to augment published content. For example, a user of
a global map facility could use a general annotation facility to report
new roads in a new subdivision or inaccurate addresses; a soldier in the
field could report that a situation has changed, for instance, a bridge
is temporarily disabled. If system applications like DBMS,
workflow, or collaborative authoring are open for extensional services
like annotations, then soldiers might use publicly available information,
annotating it to adapt it to military use, instead of having the military
invest in its own tool sets to get the extensional capabilities it needs.
The technology objective of the OSA/Annotation project
was to develop an open, scalable annotation service that can be tailored
to accomodate a range of custom uses. The Annotation
Service allows existing web content to be augmented or filtered using third-party
information, which can be much richer in structure and content. Such
information adds value by elaborating on, restricting, providing a background
or rationale to, branding, and personalizing the web content based on user
The infrastructure objective was to understand
how to add new services to standard web engines in the same way that we
can for ORBs.
The technology transfer objective was to complete
a prototype toolkit available to the DARPA AITS application community,
the Web vendor and user communities, and the Web standards community W3C.
The architectural elements of a basic Annotation
An open Annotation service involves a rich design
space. A design space describes, for each part of the architecture,
the range of alternative design choices that can be supported. The
design space can be viewed as a formal AND/OR graph structure in which
AND layers describe required components and OR layers describe tailoring
alternatives. The design space enumerates the applicable alternatives
and tradeoffs, can identify the basic architectural structure of the service,
can support tailoring of the service, and can be used to generate specifications.
We found little prior work describing the design space for Annotations,
and wanted to explore alternatives that seemed promising but had not yet
been tried. For our purposes, we describe the design space informally as
a hierarchy of objectives and requirements.
annotations and annotation sets - annotations
consist of source object and anchor, annotation body, and possibly other
metadata. An annotation set is a collection of annotations on a given
annotation authoring - a person or program
creates an annotation set and stores it in an annotation repository.
annotation repository - stores and retrieves
annotations given a target URL and other retrieval criteria
URL interceptor - captures URL at dispatch
annotation service extension - the core service
that wraps the normal URL access with side-effects to access the annotation
repository and compose the returned document and annotations
annotation composition - composes the returned
document with its annotations
annotation viewing - simultaneous presentation
of returned document and annotations using some style
annotation service management interface -
an API and GUI for controlling annotations (turn on, off, select options
The following is a list of the issues and alternatives we encountered
in reviewing prior work and in designing our prototypes:
annotation representation -- annotations represented
as annotation objects: <URL, anchor, type, annotation body, metadata>
annotation bodies can consist of text, code, XML,
HTML, or other formats. Some formats such as HTML can provide implicit
support for graphics and sound.
anchor points in source or target documents can be
specified by URL, tag, offset, pattern, query, or other means. A library
for finding anchor points wold be useful for applications other than just
new kinds of annotation metadata can be specified
for different purposes: author, timestamp, type, default colors,
pedigree and provenance, security, and so on.
annotations can have behavior, e.g. accumulating
statistics or votes over time
human readable format - for viewing saved annotations
available in the repository or via the viewer
recursion: annotations, themselves objects or MIME
types, might be annotated
annotations can be used for system reasons
to mark sections of a document "Secret" and hide
typed annotations with MIME types, e.g. voice.
annotations can be used to form relationships between
a source object and a target object. The relationships can have properties.
annotations may be used as views to support transclusion,
which is the virtual copying and possible caching of content while maintaining
derivation to create a relationship between two objects.
there may be a set of presentation styles to designate
annotations should obey transcopyright rules
Intermediary function: annotation selection requires
monitoring which URLs the browser fetches
also called an interceptor or transducer
may be implemented as a proxy intermediary on server,
client, or in between that simply captures URL refs
may implemented as a browser extension - however,
browsers provide different extension mechanisms, requiring different intermediary
implementations. Also, the browser may not support URL interception in
its extension mechanisms, e.g. Netscape does not allow plugins to register
for handling HTML types.
may be implemented as a Java applet calling native
code to interface with the browser
frames, combined with server intermediary to localize and de-localize all
references on page. Applets and plugins embedded in a page may have problems
accessing data from their server this way.
current solutions are unsatisfactory in that they
compromise security or require extra-browser configuration.
The side-effect action of the intermediary typically
involves retrieving annotations associated with the URL. In the OSA/Network-Performance-Monitor-Service
project, the intermediary can be used to launch a companion performance
monitor to track and report back on network weather. Many extensions
are enabled by the same intermediary mechanism (security, advertising,
advertising suppressor, filtering, counting, ...).
The intermediary/interceptor is analogous to the
interceptor or portable object adapter dispatch traps in a CORBA system,
so this is like a URL Request Broker and the web architecture augmented
with extensions mechanisms is like the OMG architecture. We need
to merge the Web URL Request Broker and CORBA so services on the ORB can
operate on data on the web, e.g., so ORB versioning can guard the web document
tradeoffs exist regarding placement of the intermediary/interceptor
on a server, on the client, or in between: client placement provides better
performance but requires configuration on the client. Server placement
adds an extra retrieval hop, increasing latency for page retrieval.
the annotation repository
can be a file, DBMS, search engine, or other get-put
associative retrieval engine
may use a parseable format such as XML or HTML so
arbitrary users encountering the repository can see some representation
of its content.
may publish metadata about itself to search engines
so it can be located
can be local to a user, shared by a work group, public,
or federated and accessible to a "community of interest", leading to individual,
group, or public annotations.
for scaling to large numbers of users, the repository
needs a replicated data store similar to Usenet newsgroups, or may perhaps
use newsgroups directly as a file system, if security is no concern.
viewing of annotation sets can vary according to
styles supported in a browser or editor and user preference. The
style of presentation can vary: in-line annotations, post-it notes, annotations
juxtaposed with content in separate frames, the equivalent of page margins
where only the annotations relevant to a page are shown in the juxtaposed
frame etc. Multimedia voice or animation might also be played during the
annotations need not be embedded in the content as
presented, thus allowing opaque information (e.g images that cannot be
analyzed) to be annotated with text and vice-versa.
presentation styles can be "active" - i.e., include
behavior that affects other parts of user environment, such as accepting
user input for ratings
annotation-based metadata can be used to create and
scalability of delivery mechanism to support large
numbers of clients and annotations and avoid information overload
support discovery of communities of interest from
groups too large to know one another: for example
support messaging to all authors of annotations in
a particular thread.
allow filtering by list of authors so only annotations
by members of the list are viewed
browser and annotation viewer should be able to retrieve
and present content in parallel on client
support community rating of annotation content and
filtering by rating on retrieval
support expiration dates - old annotations are automatically
deleted after a time
allow expiration to be overruled, e.g. annotations
with high ratings live longer
annotations should be kept small and delivered via
lightweight mechanisms e.g. IP datagrams
user-specified filters or queries can permissively
select subsets of annotations for viewing.
Security filters can restrict some viewers from viewing
filtering annotations by various criteria e.g. author,
annotations after 8/3/97, ...
active composition: have the server retrieve content
from a newsgroup etc. to form the annotation text.
delivery time of annotations can be synchronous or
asynchronous with delivery of content
provide efficient and low-cost delivery mechanisms
to enable quick delivery of many notes
"Pulling" information from foreign (e.g. e-mail and
news) repositories and presenting it as annotations allows us to treat
traditional asynchronous interaction mechanisms as annotation authoring
simultaneously arriving annotations from different
sources might be viewed as a kind of chat about the document being inspected
annotation delivery may be adaptive to user portals
scale up to support large "hot-spot" documents with
use of standard infrastructure
provide variety of architectural mechanisms as
components to intercept and annotate web content that can be used independently
or composed to address different scalability concerns
creation of an annotation set can be manual but more
commonly will be supported by an authoring environment used to create annotations.
This may involve extending a browser or editor or, for a heterogeneous
domain, multiple browsers and editors.
annotation authoring and viewing can be non-invasive;
i.e. use standard web browsers , requiring little or no modification to
current web infrastructure or content
static vs. dynamic architectures for annotation delivery.
Static architectures (whereby the system components
are pre-wired), are further divided into transparent and non-transparent
architectures depending on whether they employ a different URL space for
Previous efforts have focussed mainly on transparent
architectures where annotated content has the same URL as unannotated content.
Non-transparent architectures, which map the annotated
content to a new URL, can provide a different set of scalability benefits.
Dynamic architectures can provide support for large
communities of communities by dynamically locating and binding to repositories.
This requires supporting metasearch, where annotations are composed
with search and metadata services to match the annotation consumer
with the appropriate annotation repositories, using a mechanism similar
to that provided by the OSA/Trader.
accommodate wide diversity in information sources
interoperate with widely used legacy mechanisms of
information authoring by generating wrappers for foreign repositories such
as e-mail and news.
automate the generation of such repository wrappers
using meta-data that characterizes the repository. An example of such wrapping
is (semi)automatically figuring out the target and anchor points of an
e-mail message being processed as an annotation.
making the overall architecture extensible and customizable
The composition and presentation interfaces of OSA/annotations
are open and dynamically customizable. This allows new classes to be written
and added at runtime to support different composition or presentation styles.
These classes can be "active" and can incorporate information sources of
higher-level facilities like pedigree drill down,
argumentation, or workflow can use the OSA/annotations service as a primitive
workflow systems typically support some kind of ability
to create relationships among objects governed by different tools.
pedigree (e.g. drill-down) visualization mechanisms
and pedigree-based information filters.
Pedigree information could be a query on dynamically
changing information sources
link mechanism that is used for annotation argumentation
will be adapted to support static and dynamic (query-based) specification
of the annotation pedigree
information source certainty and believability metadata
can be accessed
support adding "active" composition, presentation
behaviors, new anchor types dynamically
call back mechanism - this mechanism allows a document
provider to associate with web pages she authors a list of actions that
can be taken with respect to the page. One such action is to associate
with the page a list of annotation repositories to associate with the page.
These would be searched if the viewer requests access to the page.
This is a server side mechanism.
Limitations of Current Work
At present, browsers have no general
mechanism for third parties to provide value-added information to the Web
and make it available to others. The state of practice mechanisms available
today requires sending email to Web page authors suggesting adding updates
or doing this more publicly if the page author includes comment boxes on
his web page, which is a very limited form of annotation.
Recent work on Web annotations includes the CERN
annotator, Stanford ComMentor, CritLink,
CoNote , Strand/GrAnT and the Net Notions product.
While the architectures of some of these systems
are open in one way or another (for instance, including the idea of public
and group annotations), the implementations tend to require modified web
infrastructures, some systems are slow, and none has been widely deployed.
CoNotes requires document authors to anticipate places where
readers might wish to add commentary or questions by inserting "annotation
points" into the original document. CoNotes can only annotate documents
under its server's control, and does not support annotating arbitrary web
content. In HyperNews and CritLink, the request for annotated content is
syntactically different from a request for the unannotated document. HyperNews
system is primarily a discussion group tool, but has supported annotation
in conjunction with the use of modified browsers (Mosaic and HotJava) in
the past. CritLink uses a specialized server called the mediator. The Mediator
retrieves a requested page, prefixes all references in it with its Internet
domain, and returns the annotated, modified page to the user. All further
navigation from links within the returned pages are forced to be processed
by the Mediator. This approach requires users to navigate from within the
CritLink-processed page instead of by using the browser's bookmarks or
direct URL entry. Both NCSA's Mosaic project and ComMentor modify web browsers
to augment them with annotation capabilities. The NCSA Mosaic browser supported
private annotations and group shared annotations in several releases. Both
Strand/GrAnT and the Net Notions product introduce web annotation function
without modifying web content, browsers or servers. They are similar in
philosophy to the systems that we have built. Strand uses a client intermediary
for page interception. The Strand authors noted some problems with changing
page layout when inserting annotation indicators and functions, and envisioned
that much of the GrAnT functionality might be moved into a Java applet,
the approach that JotBot explores. Net Notions is a commercial product
from Sideware Systems, Inc. that allows users to affix text annotations
"over" arbitrary web pages. Net Notions uses a client process that apparently
records current URL and position information when annotations are created.
Other topics related to annotation include:
-- the Xanadu notions of virtual inclusion, "which permits composite
documents to be constructed by reference to the original components rather
than by copying" including permissions for such references.
link services -- these services explicitly
remove references from the original source document and store them separately
so that URLs are not embedded within documents. This allows the same
document to be annotated bidirectionally with different collections of
links for different purposes.
relationships -- annotations can be viewed
as a kind of relationship. OMG has defined a Relationship
Service for connecting distributed CORBA objects. The OMG Telecom
Task Force has issued an RFP for a Topology
Service for managing graphs of relationships (see HP
rating services - PICS
and now PICS-NG respectively support
the addition of metadata for rating purposes and for more general purposes.
These are related to annotations in that annotations can themselves be
viewed as metadata, annotations can use metadata schemes, and conceptually,
URL access is overloaded with an additional behavior, to filter or negotiate
changes to content to match viewer attributes so a similar interceptor
scheme can be used to install PICS into a browser.
interceptors -- OMG has defined two mechanisms
for dynamic dispatch to over load the normal CORBA IOR access mechanism,
one is the interceptor from the Security specification (Part
I and Part
II), the other the Portable Object Adapter (Part
I and Part II).
Jigsaw servers support server-side before and after interceptors.
We have constructed two prototype OSA/annotation implementations to illustrate
these different aspects of an open annotation service. Early releases of
both prototypes were demonstrated at the DARPA PI meeting in San Diego
during 10/97. A reference model for annotation architectures is presented
in a paper to
be published in the HICSS '99 conference. The Server-side prototype
modifies the Jigsaw web server and maps annotated URLs to a new URL space.
The JotBot prototype takes a client-side approach by using a Java applet
to retrieve and compose annotations within the browser, allowing the annotation
server to be thin.
The Server-side Prototype supports:
The JotBot prototype supports:
an Annotation class supported by the annotation repository
(implemented using a Java persistent object store)
URL descriptor objects - these are container objects
that contain annotations for a particular URL
annotation bodies -- can be type text or HTML.
oid -- a system-assigned oid for each annotation.
Used in link modeling.
anchor types supported -- The anchor is either URL
or positional. Positional tags are used for embedded annotations.
metadata -- currently a timestamp field is supported
as an example of metadata.
Links that allow annotations on annotations
Link objects are a triplet (source annotation, target
annotation, link type)
The Link data structure can accommodate multiple
link types, and many-to-many links.
Link navigation methods are supported.
Currently, links are scoped to annotations on the
Annotation repository wrappers - The annotation repository
has three wrappers that support the same annotation API with different
The Java wrapper returns query results as Java data
structures. This wrapper can be used by clients that are co-located with
The HTTP wrapper supports queries specified as URL
fetches (like CGI). It returns query results as human viewable HTML pages.
This wrapper can be used by clients anywhere to query the annotation repository
annotation creation -- via an HTML Form-based interface
annotation access -- overloads the Get operation
in a Jigsaw server - requires the user to use a different URL for annotations
than for the original document - The prototype uses jigsaw facilities to
associate the two URLs.
two presentation styles
annotations juxtaposed with content in separate frames
in-lining annotations at structural or positional
anchor points. Structural anchor points are demonstrated by the use of
existing HTML tags as an anchor for in-line annotations.
OSA/Annotations supports audio annotations as is:
HTTP servers already have the ability to publish
these files as URLs which are replayed by client-side helpers/plug-ins
in the usual way.
A hyperlink to audio/video shows up embedded in the
doc in the usual way, and the helper tool plays the audio when the hyperlink
We can author annotations without requiring any change
to web infrastructure since the editor and server are independent components.
We can retrieve and present annotations also non-invasively since the Viewer
applet is an independent component. The viewer applet needs only to know
when the browser fetches a new URL, and it has an open interface for a
browser-dependent interceptor to use. The Survey, Rating, and Messaging
functions are innovative in that they adopt features found in other collaboration
research for use in an annotations context.
annotation bodies -- Java objects, which could be
built dynamically from arbitrary other types, e.g. ontologies, at retrieval
time. The code for the "Compose" and "Present" methods is loaded dynamically.
anchor types -- keyword for text match within the
Unidirectional link to a URL or image (meaning determined
by loaded behavior)
name of Presenter class
name of Composer class
color to be used by presenter
simple editor applet - working and integrated with
viewer. Currently, the editor occupies one browser window, while opening
a second browser window for navigational use.
editor is a Java V1.0 applet that runs in a browser.
the editor controls annotation size (so they can
be delivered as datagrams) and does simple content validation
annotation access -- URL interceptor implemented
using a Java native-code library
this is now working for demonstration purposes for
Netscape only, although the library supports IE as well and the browser
restriction can be removed.
annotation repository --
multi-threaded server, can be evolved to use datagrams
instead of maintaining IP connections
lightweight - process keeps no connection state so
it can service more requests
supports small annotations, sent asynchronously with
respect to retrieval of target page
annotations have a default lifetime after which they
are deleted, but accept community content ratings. High ratings extend
the life of an annotation.
allows filtering of annotations by security clearance
and author (demoed in release 1, 10/97, later disabled)
can also support delivery of larger annotations via
an IP socket if this quality of service is needed
currently implemented on file but could be migrated
to use DB or other store.
messages to discovered author community
send SMTP email to list of annotation authors including
reference to annotated page
dynamically loaded presentation styles
several Presentation behaviors were implemented under
the same interface
LinkTo - presented as button that causes main browser
to display new page of specified URL, in effect allowing new HREFs to be
added as annotations to existing pages.
audio note that plays .wav file on button press (demoed
in release1 10/97, later disabled)
image note - displays a .jpg image (demoed in release1
10/97, later disabled)
Survey - has behavior/GUI to accept votes pro/con
for user-supplied questions
Rating - accepts user votes to prolong the life of
a given annotation
viewer - displays URL content and annotations
It is a restriction of current browser implementations
that they don't publish URL fetches, and getting around that restriction
- even to a limited extent - has proved difficult. By introducing a client
intermediary we can obtain most of the information we need and modify content
sent to the browser arbitrarily without being limited by browser security.
We should not need to modify infrastructure to get URL info if the browser
allows access to it in a way that supports its security controls. It would
be desirable for browsers to provide more open interfaces in the future.
During late 1997 and 1998, XML became a hot topic
and gained significant momentum as a future web technology that will ultimately
complement - but not replace - HTML. XML's support for semantic tagging
makes it possible to express annotation objects and links as XML elements.
The use of XML could potentially make most web documents accept annotation,
and facilitate development of more powerful forms of annotation. An XML-based
annotation repository could offer a lighter weight alternative to other
data store choices. However, a number of concepts in XML such as semantic
tagging and the ability to create external hyperlinks between documents
could change the architecture of browsers, and therefore the architecture
of annotation systems. The XML developments occurred too late in this contract
to really allow time to redirect our work to focus on XML. At present,
standard browsers do not yet support XML and the nature of this support
remains to be seen. HTML is likely to constitute the vast majority of web
documents for some time to come. Given our goal of supporting commonly
available technology unobtrusively, we decided to continue to work with
existing browser technology.
During the spring of 1998 we worked closely with
a local entrepreneur who wanted to license the annotation technology for
use on a future commercial web site, as a means to conduct online focus
groups and reviews of business plans. This effort gave us some positive
feedback about usefulness and usability of the tool as well as providing
insight to the work that would be required to deploy the software for commercial
use, testing on all browsers and server platforms.
Also notable was the release of the Net Notions
product by SideWare, inc. in 1998. This annotation product was the first
to appear commercially, and it embodied a design similar to the novel approach
we had taken in developing JotBot, using a client process instead of a
Java applet to monitor browser activity and retrieve and present annotations.
Although NetNotions was built by a larger software development team, it
underscores the shortened time it takes for an idea to go from research
to commercial product using today's internet.
A general mechanism for annotating web documents
can be the basis of a number of new and useful document management applications.
We have developed an architecture to unobtrusively annotate web content
and two concrete implementations of this architecture. Useful annotation
capabilities could be introduced into today's Intranets and Internets without
resorting to specialized infrastructure, by placing a simple intermediary
function either in the browser (perhaps as an applet, as we have demonstrated)
or at the server and controlling access to the intermediary. However, some
relatively minor changes in current browser and server technology would
make it much easier to extend their functionality. We have identified several
desirable changes on each end, and would be gratified to see commercial
products that adopt a more open stance toward extensions.
These conclusions are also summarized in our HICSS
'99 conference paper "On Web Annotation: Promises and Pitfalls of Current
The technology we have implemented is now released.
Directions for further work include
build vertical applications on top the core annotation
function, e.g. by constructing a spider that automatically creates annotations
to enhance or update information available on various topics.
address scalability issues associated with large
scale annotation systems:
finish extending the server software to use datagrams
instead of IP connections
modify the data store to use a commercial DB instead
of a file
modify the data store to use newsgroups for storing
annotations to get wide-area replication
explore using Trader
architectures to dynamically locate and access appropriate annotation repositories.
Communities of annotation users having similar interests but not be known
to each other. An OMG trader-like service might be useful to match an annotations
client to a relevant group's annotation repository.
The OSA/Annotations architecture allows the stored
form and the presented form of an annotation to be different. As appropriate,
the system may lose information or transform the contents to a different
evolve the capabilities of the systems we have built
to exploit evolving web infrastructure.
define a generally flexible and extendible annotation
structure using XML
use XML to build the annotation repository
using an XML browser, find ways to integrate annotations
for non-XML pages with those for XML pages.
is sponsored by the Defense Advanced Research Projects Agency and managed
by the U.S. Army Research Laboratory under contract DAAL01-95-C-0112. The
views and conclusions contained in this document are those of the authors
and should not be interpreted as necessarily representing the official
policies, either expressed or implied of the Defense Advanced Research
Projects Agency, U.S. Army Research Laboratory, or the United States Government.
© Copyright 1997, 1998 Object Services and Consulting,
Inc. Permission is granted to copy this document provided this copyright
statement is retained in all copies. Disclaimer: OBJS does not warrant
the accuracy or completeness of the information in this survey.
Last revised: September 15, 1998. Send comments
to Venu Vasudevan or Mark