WebTrader:  Agent Discovery

Project Summary

Tom Bannon and Craig Thompson
Object Services and Consulting, Inc.

17 June 2002


Executive Summary

WebTrader is a lightweight trader/matchmaker/yellow pages that is scalable to WAN environments and can be used to locate Internet resources (e.g., services, agents, data sources, search engines, other traders, anything) whose advertisements can appear on any web page, represented in XML.  WebTrader depends on existing search engines to index the advertisements and effectively provides a meta search front end that can locate Web pages containing advertisements that match its query.

We demonstrated evolving versions of WebTrader at CoABS Workshops at Las Vegas (January 1999), Northhampton (June 1999), Science Fair in Arlington (October 1999), Atlanta (February 2000), and Boston (August 2000).  Thereafter, we halted the project, per Jim Hendler's guidance, to provide added resources for eGents and MIATA, CoAX and JBI TIEs.


The objective of the WebTrader project is to develop WebTrader Query Tool (aka WebTrader), a trader/matchmaker/yellow pages that can scale to WANs and permit anyone on the Web to advertise a resource (e.g., agent, service, data source, search engines) that anyone else can discover.  The approach to scaling is to represent advertisements in XML, store them on Web pages, let existing already pervasive and industrial strength search engines index these pages, then WebTrader's engine accesses one or more search engines, locates pages with advertisements, matches the ads against the request, and returns the matching advertisements.  One interesting application of WebTrader Query Tool is DeepSearch, a recursive/federated implementation of WebTrader that acts like a normal search engine but locates search engines at local sites and recursively searches these.  Many web search engines only index top-level pages leaving the other half of the web unindexed except by these local search engines.

Specific scientific and engineering subgoals were:

WebTrader alone is not an agent system, but it is a generalized component capability of most agent systems, providing matchmaking.  The interesting aspect of WebTrader is its scalability to the Web.  Seen in this light, WebTrader can be viewed as a standalone agent grid enabling component, but one that operates in open WAN environments, is compatible with not only agent but also object and ontology technologies, and has no downloading barrier to widespread deployment.

Technical Accomplishments


The basic architecture of WebTrader is fairly simple.  Essentially, WebTrader is a new kind of specialized meta search engine that wraps other search engines to return typed advertisements from the open Web.  As shown in the figure below, any page on the Web (1) can be annotated by anyone with an XML WebTrader advertisement (2) - see example Trading Advertisement DTD.  These pages are indexed (3) by ordinary Web search engines.  When a Query (client advertisement) (4) is presented (5) to WebTrader's engine (6), it accesses known Web search engines (7), they find matching webpages in the usual way (8) and return then (9).  A WebTrader matching algorithm (10-12) finds candidate advertisements, scores them, and returns a series of responses in rated order (13-14).

Internet Resource Advertisements, Trader Federation and Rebinding

Our Salt Lake City demonstration of the WebTrader (January 1999) illustrated: A significant evolution in the design of the WebTrader ads occurred along the way, as we realized we could rework advertisements so that resource description in ads could be independently defined, allowing anyone to create an ad for anything they cared to define or use an existing definition of, from CORBA IDL components to bicycles.  This change also affected ad matching and returned results. The revised TradingAd DTD now defines <!ELEMENT resource ANY>, allowing resources to be any XML document, from reconnaissance reports to parts inventories, from Java RMI components to Oracle databases.  This successfully decoupled resource definition, basically an ontology question, from the WebTrader design, a major step forward.

We also made progress on service binding and rebinding and trader federation.  In the Salt Lake City demo, trading ads exist for a variety of components (e.g., service agents, clients, and WebTraders). Metadata including color and cost is included into the ads.  A Blue client, for example, asks the USA WebTrader to locate a Blue agents implementing a particular interface and with zero cost, if possible.  One is found and bound, and the client makes use of the agent.  When the agent unexpectedly dies, the WebTrader is consulted again by the client, in case the “state of the world” has changed.  It gets back a new list of agents, sorts them by cost, and goes down the list trying each agent until it finds one that works.  If the WebTrader fails to respond, the client can fall back on its cached list of previously found agents. When the Yellow client asks the USA WebTrader for Yellow agents, the WebTrader’s initial search turns up none, as it only consults a domain that indexes ads of Red, White, or Blue agents.  However, it does find an ad for a WebTrader that knows about Yellow agents, and so passes on the original client query to the Euro WebTrader, which finds a Yellow agent, passes it back to the USA WebTrader, which passes it back to the client, which then uses it to connect to the agent.  At the right in the figure is shown a service advertisement in XML.  For more detail, see WebTrader Demo Script.

WebTrader Query Tool

Our status in Salt Lake City was, we had a better understanding of both webtrading and also the deep search problem but separate implementations.  The next and main phase of the WebTrader project was to develop WebTrader Query Tool, a re-designed and re-implemented next generation WebTrader/DeepSearch capability that combined and generalized all functionality in a single prototype.  This next generation WebTrader was engineered so that the front-end works as an applet with good performance on most browsers (tested on Internet Explorer and Netscape) and the backend is a servlet.

In a demonstration from the CoABS Boston Workshop (August 2000), WebTrader Query Tool was coupled with a modified Java App Builder (a visual programming tool), extended with WebTrader drag-and-drop capability.  In the demo, the user creates a want ad called ColorButtonsSpec.xml, drags it to WebTrader Query Tool, runs a query which returns pages containing ads for colored buttons, then selected buttons are dragged to the WebTrader App Builder and connected together to build a simple application.

WebTrader Query Tool is a generalization, redesign, and reimplementation of previous work in numerous ways -- the more significant changes are listed below.  Almost all code was rewritten from the Salt Lake City implementation.

Gridifying WebTrader

The CoABS grid is a JINI-based implementation of an agent interoperability platform developed by GITI, the DARPA CoABS program integration contractor.  It is an important, on-going experiment in agent system interoperability.  As described elsewhere, we contributed architectural ideas to the grid.  But in addition, we developed three standalone agent components (eGents, WebTrader, and AgentGram) that can play a role as (stand alone or connected) grid components or services.  As part of the Agility WebTrader project, we developed the grid-relevant capabilities described below.  At the same time, we note that WebTrader is itself a potentially pervasive stand-alone grid capability that can supply matchmaking for agent, object or ontology systems.

For the Science Fair (October 1999), we gridified the WebTrader Agent (WTA).  Previously, in NEO TIE #2 (see description below), the WebTraderAgent (WTA) was used  as a WAN matchmaker to provide SRI OAA and USC/ISI Ariadne the ability to dynamically extend its information sources based on user queries it received.  For the Science Fair, we modified the WTA to query the Grid for any WebTraders registered there.  If none is found (or timeout), then the WTA automatically falls back to its local embedded WebTrader.  This essentially extends CoABS grid JINI lookup service to be a WAN lookup service.  The implementation involved creating the WebTrader Grid Service (WTGS), a Java program that registers a WebTrader on the CoABS Grid and logs its actions, along with a new version of WebTrader Agent (WTA) that lists WebTraders found on the Grid.  We worked with Hank Seebeck and Adam Wenchel (both GITI) to test WTA on the grid.  For the Boston CoABS Workshop (August 2000), we upgraded WebTrader Query Tool to use the grid in the same way as shown in the diagram below.

We also created a launch page for a CoABS 7x24 grid accessible version of DeepQ.  WTA was maintained on the CoABS 7x24 grid for approximately a year.

Technology Transition

In 1999, we applied WebTrader in the DARPA CoABS Non-combatant Evacuation Order (NEO) Technology Integration Experiments (TIE).  The NEO TIE involved an urban rescue effort and served to organize many CoABS program activities in the first year of the CoABS program.  The thrust was on agent interoperability and rapid assembly of heterogeneous agent systems to solve problems.  Lessons learned and components from this exercise where later incorporated into the CoABS grid.  TIE #2 involves the scenario Find Civilians, Get Them to Embassy, and the interaction scenario below shows not only what information is required but also the role of various CoABS subsystems:  OBJS WebTrader, OBJS AgentGram/MBNLI, ISI Ariadne (which extracts information from Web pages), and SRI Open Agent Architecture (OAA, which enables inter-agent communication).  Steve Minton (ISI) coordinated the TIE.  For the NEO TIE, we developed WebTraderAgent, an OAA wrapper for a WebTrader.  This effectively added a Web-wide trader to OAA.  In the demo, when a new service or data source is needed, the WebTraderAgent is consulted to locate a WebTrader which can then be consulted to locate advertisements for matching services.  In addition to building the OAA wrapper, this involved porting WebTraderAgent to JDK 1.2 (Java 2), creating a stripped-down security policy for WebTraderAgent, and using an executable jar file to house the agent.

In March 2000, we presented work on Agent Discovery Matchmaking to OMG, essentially, a sketch of a reference model for traders that subsumes WebTrader as well as conventional agent matchmakers and LAN traders like OMG's Trader.

Next Steps

The main puzzle preventing widespread use of WebTrader is the problem of populating the Web with advertisements.  Since advertisements can be about anything, this is essentially the ontology problem.  The entire DARPA DAML program as well as W3C RDF and semantic web communities are working this problem.  Universal Description, Discovery, and Integration (UDDI) has recently been working to register businesses and their services.  Finally, various Want Ad repositories have developed interesting proprietary ontologies.  Since WebTrader's architecture is relatively independent of resource descriptions, the architecture can use any or several of these ontology representations.  Both DAML and UDDI are examples of manually creating advertisements.  An alternative is to automate this process (insofar as possible).  A potential source of ads is the Web itself, namely, mining Web pages for ads of interest, especially in tractable domains.  Alta Vista has done this with various easy-to-recognize data types like images and mpg's.  It might not be hard to find certain other kinds of elements, for instance, banner ads.  Similarly, our DeepSearch project has developed ad hoc recognizers for locating local search engines on Web pages.  Of course, both manual and automated means of populating the ad space would be useful.

It is worth noting that the basic design of WebTrader appears to be that of an open world repository, that is, ads can come from any pages that search engines index.  Interestingly, it is not a large change to convert WebTrader into a closed world search engine only containing ads that are inserted into its database (instead of SE index).  This might make more sense for some kinds of services, e.g. resume services or a company's web services, where the service provider might want control over who can advertise.

Although robust in several ways, the WebTrader implementation is still a prototype.  More work is needed in several areas:

This research is sponsored by the Defense Advanced Research Projects Agency and managed by the U.S. Air Force Research Laboratory under contract F30602-98-C-0159. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency, U.S. Air Force Research Laboratory, or the United States Government.

© Copyright 1998, 1999, 2000, 2001 Object Services and Consulting, Inc.  All rights reserved.  Permission is granted to copy this document provided this copyright statement is retained in all copies. Disclaimer: OBJS does not warrant the accuracy or completeness of the information in this survey.

Last revised:  October 2001.  Send comments to Craig Thompson.

Acknowledgements:   Paul Pazandak did most of the design and implementation with Craig Thompson providing brainstorming and review.  Steve Ford installed and demoed MBNLI at CoABS Workshops that Pazandak did not attend.  Thompson completed the CoAX TIE demo.