Services architecture

• What sort of architecture is appropriate for DDB?
• What is a services architecture?
• What services does DDB need?
• Architectural Issues

What sort of architecture is appropriate for DDB?

• traditional database architectures (relational, object, object-relational)
  • all the data is inside the DBMS (owned by the DBMS)
  • there is usually one supported DBMS data model
  • the DBMS exposes a fixed set of functionality
    • so you wrap it with additional functionality you need
      • line of sight
      • uncertainty
      • time
      • middleware functions
      • workflow
      • many more
    • these are all add-ons - since internal DBMS APIs are hidden, inefficiencies will result
• for the DDB
  • the data is collected in many places in many formats in many quantities
    • not all data is owned by one owner so there may be cost and legal boundaries surrounding some data
    • not all data is processed by one owner so there may be firewalls and security perimeters around the data and functionality
  • the data is analyzed in many places so the analytic functions as well as data are distributed
  • the data is aggregated in many places
  • there are many consumers with diverse needs for views of data (many products)
    • the field of requirements is greater than any one system with fixed functionality can handle
  • geospatial data is not the only data of interest
  • but there is still a need for a consistent global image of what data is available where and in what form to coerce it into the form you need when you need it
  • in fact, there is a notion that a geospatial index is one of the fundamental ways to organize information and should underlie (overlay) not only DoD C4I/logistics/*int but also the web of knowledge
  • it is hard to say where geospatial data ends and application data begins
    • scoping issue is, do we use the same representations for digital battlefield maps as for agricultural uses, do we need maps just of the airspace or also the meterology, what about oceans and petrochemical deposits, what about simulation
• lesson learned from engineering DBMS'
  • central representation more important than central data storage
• hypotheses
  • the DDB must be distributed both in data and function
  • the DDB architecture must be open so new functions as well as formats can be added
  • the DDB may be more a framework than a closed, centralized, monolithic system
• trends
  • open, extensible DBMS systems see other foil & mention OLE DB
  • open oo middleware architectures reference OMG, ActiveX, Java, …
  • accommodating diversity via interoperable protocols in IETF and web
    • web + OO + security + transactions + caching + … global OODB?

What is a services architecture?

• old way - define a system architecture as a fully connected collection of boxes with in-out connections to each other forming a call graph
  • hard to add new boxes or change connections, which are hidden to outsiders
• architecture desiderata see other foil
• services architecture - define architecture as message passing bus with services hanging off. Services expose interface, implementation is hidden.
  • design principles include separation of concerns and projecting out capabilities
  • can hook up functionality different ways
  • can add new functions to running systems
  • very extensible - can add services, replace or refine services, add policies to govern services
  • seems scalable - via composition and federation - remains to be proven
  • object services architecture (OSA) - use OO IDL like OMG's to specify interfaces
  • similar idea to frameworks
    • benefits end users - common semantics >> common look-and-feel
    • benefits developers - rapid application development based on reusable parts (or so the story goes)
• many open questions
  • are services' implementations orthogonal?
  • can I compose systems from services - how to hook up the parts
  • are composed facilities more interoperable if composed from similar services
  • how to define and enforce boundaries, scope, perimeters
  • can I throttle - move objects and processing around?
  • what is not an OSA? How much is really explained by these architectures?

What Services does DDB need?

• Representation
  • data formats
  • GURL/TURL - geospatial and temporal URL - (semi) universal way to reference a location and time
  • common schema, many schemas
  • extensible schemas and weak schemas for semistructured information
  • metadata
  • relationships, annotations, pedigree, uncertainty representations
  • behavior, events and rules
• Transformation and Analysis Services
  • sensor/pixel to entity image understanding …
  • algorithms sensitive to data type and granularity
• Communications, ORB and middleware, dataflow and workflow
  • long list from OMG presentation - cost-based QoS, streams, events, persistence, transactions, queries, notification, …
  • non-trivial intersection with DBMS services below
  • Security Services …
• Database Services
  • persistent storage
  • indexing
  • caching
  • replication
  • distribution
  • recovery
  • storage hierarchy
  • concurrency and transactions
  • query
  • optimize
  • metadata catalog
• I*3 Services
  • wrappers
  • domain-specific indexing
  • mediators
  • annotations
  • weak object models
  • ontologies, semantic integration
  • subscription/notification
  • query relaxation
  • mobile agents
  • rules
  • versions
  • explanations
  • certainty
  • argument structures
  • pedigrees
• Collaboration Services
• Visualization Services

DDB Architectural Issues

• Scope of DDB problem
  • accurate representation of battlespace, past, present, and future
  • model the entire planet to finest grain anyone would ever want
• General shape of the architecture and architectural boundaries
  • conceptual/functional boxology, not physical architecture
  • DDB at center of picture, one big box, mother of all DBMSs -versus- distributed consistent framework with mediated control over network of information sources
  • what won't DDB do? Is it a MIDB replacement or augmentation?
  • unification versus accommodating heterogeneity
  • What functions go inside the DDB? is there an inside?
  • Where is the data, who owns it, what formats, is most archived or active, retention policies, shape and contour of the data size (sizing analysis) and location problem. Is it enough to catalog most data so you know how to get it, must it be inside the DMBS?
  • What are the functions and where are they located? Cost model of what information to collect, store,
• Must we build the following in early - or are they modular additions?
  • security
  • real-time
  • managed object interfaces
  • QoS and optimization - many producers and consumers - optimization for one might have tradeoffs for others
• Missing OMG services
  • Rules, Versions, Indexing, Information Source Wrappers, Caching and Replication, Asynchronous Messaging, Visualization, Annotation Service,
  • existing and future service implementations not available from any one vendor

Missing Ingredients in our story

• Who is customer of DDB? NIMA, DoD, industrial users, web
• What is the problem we want solved?
• Glossary of terms -
• Current state of practice - how do we do it today?
  • status of DDB-related industry
  • status of DDB-related standards
  • status of target environment - DoD process today
    • politics - separation of strategic from tactical tools
  • how DDB relates to other DARPA programs
    • technology programs - I*3, IC&V, …
    • application programs - JTF, BADD, JFACC, ALP, …
• Tech transfer approach - at least as important as developing technology and architecture