We OPML'd the DoD

I'm just giddy about this. We OPML'd the DoD. Specifically, we published OPML for Joint Capability Areas. It turned out to be so easy, too. I wish I could easily convey the significance of this achievement.

JCA's are an exhaustive military taxonomy declaring hundreds of defined military capabilities. They are predominantly referenced in decision making for acquisitions and planning. All over the military there are organizations who are required to map every aspect of their existences to JCAs or else be denied resources. Consequently, nearly every “decision support” and “business intelligence” tool under the sun (there are hundreds, if not thousands of them; costing tens, if not hundreds, of millions) has a requirement for mapping JCAs. Just this week (in fact, after we decided to do this) one of our teams received a requirement to add JCA mappings in a database. The first question that came to my mind was, "how are they going to get that data into the system?"

You see, JCAs are published in formats that are only readable by people, and not in formats that can be used by tools. As a result, people repeatedly spend unnecessary time manually keying in JCAs just so they can be available for menu items, pick lists, and other input forms in their software applications. This is inefficient, error prone, and easily solved by publishing JCAs in Outline Processor Markup Language

So that's what we did.

Now system integrators can consume a URL-addressable, well-formed, hierarchical, text document as input to their application development. No more need to manually key stuff. No more typos. Point any developer worth his/her salt as this URL and they will be off to the races.

It's too bad the owner of JCAs doesn't publish more portable content, but maybe now they will.

Credit where it is due:

Our (2nd) wonderful co-op student from Rochester Institute of Technology, Ben Kaiser wrote the code. Longtime Bridgeborner, Rob Shell shared some SME-ness and was Johnny-on-the-Spot with this (mostly well-formed) HTML version of the JCAs. Thanks also to Sunlight Labs for feedback on choosing the right technology to get the job done. Raymond Yee provided an excellent model for us to follow (in response to this project idea from Clay Johnson).

Questions I Should Be Able to Ask My City (On the Web)

It's too hard to ask a simple question about a city and get a simple answer.

This notion goes well beyond satisfying the need to know the answers. Our inability to ask basic questions about any city and provide simple answers is a much more profound shortcoming compared to just the answers themselves. Simple answers to basic questions are the foundational elements of more complex analyses.

When we rely on experts and representatives, elected or otherwise, to perform these analyses we expect accuracy and completeness. But how is this measured? No matter what the field of study, the answer has always been references, data sets, and peer review.

Today, especially in matters of public interest, the word "peer" is hard to define. All the more reason why satisfying the first two elements, references and data sets, is of paramount importance.

Cities should provide simple answers to basic questions.

Cities should provide simple answer to basic questions in a URL-addressable, well-formed, text format.

There are many mechanisms cities can use to satisfy this requirement; Web API, online spreadsheet, posted XML documents. Ideally, technology exists and can be integrated so that cities can answer questions via several of these mechanisms.

So what are the questions we would ask our cities, if we could? Following are a few to which I would like to know the answers. Moreover, given the answers to these questions in URL-addressable, well-formed, text format(s) the answers to these questions could make for some rather interesting, useful mash-ups.

1. Where is the location of each road and intersection under construction in my city?
2. How long has road/intersection X been under construction?
3. When will road/intersection X construction project be complete?
4. What is the list of each building construction permit in my city?
5. Where is the location of each building construction permit in my city?
6. What is the list of businesses that pay taxes in my city?
7. What is the list of businesses that pay taxes in my city, by type of business?
8. What is the list of how much each business paid in taxes in my city?
9. Where is the location of each business that pays taxes in my city?
10. What is the list of neighborhoods in my city?
11. How many houses are in each neighborhood in my city?
12. What is the list of schools in the city? (This should include private and church schools, too, if these schools are required to be registered with the city.)
13. What is the location of each school in my city?
14. What is the enrollment at each school in my city?
15. How many cars are registered in my city?
16. What is the list of most traveled streets in my city?
17. What is the list of traffic accidents in my city?
18. Where are the locations of each traffic accident in my city?
19. How many pounds of garbage are collected in my city?
20. How many pounds of garbage are collected in my city, by zipcode?
21. How many pounds of garbage are collected in my city, by neighborhood?
22. What is the list of events in my city?
23. When are events occurring in my city, by time period?
24. Where are events occurring in my city?
25. What is the list of conditional use permits in my city?
26. What is the list of all emergency response stations (fire, police, medical) in my city?
27. Where are the locations of each emergency response station in my city?
28. What is the number of emergency calls in my city, by time period?
29. What is the list of stations responding to emergency calls, by time period?
30. What is the frequency of emergency calls by station and time period?

[To be continued...]

Without a doubt, this list is far from exhaustive. Crime data is one of the most popular and reported-on data sets. I tried to come up with relevant questions for which data are not likely accessible. Certainly, schema are required for declaring the answers to these questions in our required format. Probably every question does not have to be anticipated a priori. We should also not assume that a particular question implies a specific data set. We often can easily get the answer of, for example, the location of each crime. Rarely, however, can we get aggregate data, useful in trend analyses. Perhaps what we need are structured data sets and Wolfram Alpha. In any case, a foundational platform of data is required. In any case, we should be able to formulate a basic question and get a simple answer (on the Web).

Given the simplicity of what is being asked, however, this should not be complicated.

Wiktionary

Update: 12.03.09 - It seems thefreedictionary.com is already providing most of what I'd like to see from a service: anyone can add (don't even have to auth), domain attributes, links to references, direct URL to entry: http://acronyms.thefreedictionary.com/ There are some obvious difference in approach. I want to go further with URLs. I'd like to be able to point to a unique defintion via a path, ex., /wiktionary/FOO/1. I also want to do lookups in other dimensions, ex., return all terms from domain:X.

The government is awash with acronyms. New acronyms are created daily. Acronyms create a barrier to understanding if they cannot be easily resolved, where easy = universal and universal = URL. There are many online dictionaries with entries that are found in Web searches. However, these return results only in highly formatted, not-well-formed HTML that is not always accessible through simple URLs. Furthermore, these dictionaries provide no way for the community to create and share new entries as they are needed. A simple solution to this is:

1. Use the cloud to store terms and definitions
2. Use Web services to return definitions through URLs as XML, JSON, and XHTML
3. Provide a simple form that lets registered users add and edit terms

We have created a proof of concept here using XAMPP and AWS:
http://dev.os.bridgeborn.com/wiktionary/

It's not perfect or even complete. For example, the XHTML returns errors from the W3C validator. But I think this is a solid start and I'd like to open it up and see it go further. To that end, I submitted it as an idea for Sunlight Labs. We'll see if it garners any votes of interest...

Pentagon Internet Utility Model

Pentagon gets it right:

Now that you have a space in your cloud to put all your videos, you can come up with much simpler plans for multi-level security in that cloud using technology means that have been thoroughly vetted for decades.  Better results with less technology burden.  This can solve nearly any of your needs, not only the just-in-time requirements driven by the popularity of personal video production and sharing.  Certainly every unclassified activity is covered and that alone makes the world better by several orders of magnitude.  But, replicate this model on NIPR, SIPR, and JWICS and every possible thing you could want to do with digital video will be covered by (roughly) the same technology solution.  People who have to manage security layers will still have to manage security layers, but that concern is more easily separated from the entanglements of technology now.  This is a good thing all around in ways I hope and think you realized before I ever commented here.  

This is why you're hot.

More thoughts on Federal Agency Internet Utility Model.

P.S. "OneSource" is a bad term.  When it comes to shared content, there is never one source.  That's the point.  You are the infrastructure, just need to figure out how to make that catchy for PR and marketing.  One Destination, maybe.  ...needs work in any case.

A Portable Language for Real Time, Interactive, Rich Media

Following is a draft section from an in-progress whitepaper on ETL-V:

The key to enabling visualization as a core capability is robust description of functionality that is machine portable and parsable through a universal processor that enables the serialization and deserialization of any object, attribute, and behavior in a standard way. Such a description captures the language of visualization in a way that can be understood by any program or person.

Our machine portable language is not intended to express low-level artifacts like vertex points and surface normal vectors, although it can. Most often we follow the model of HTML, which uses URLs to reference images, movies, and other digital multimedia. A 3D Model is not unlike an image:

<img src="images/car.jpg" />

<Model url="objects/car.lwo"/>

Moreover, our language expresses high level artifacts that describe things a person without prior subject-matter expertise might easily recognize. Camera, Model, and Light are but a few examples of rudimentary artifacts in our language. Yet, to establish a robust platform for visualization we must be able to describe artifacts of greater complexity. We must describe the attributes of, for example, ColumnChart, AreaChart, FlowChart, TreeMap, and RadialTree.

Some objects have implied behaviors such as a Sequencer that iterates through a list of items or a Directive that traverses a graph.

Our language must contain more than nouns; it must also contain verbs. Therefore we have command objects like Append, Remove, Interpolate, Morph, Notify, and many others (see also Command schema, below).

Our language must be capable of describing attributes having multiple dimensions, for example a 3-dimensional position. We classify attributes as belonging to one of four basic categories: primitive, complex, compound, and collection. These are defined as follows:

1. Primitive Attribute – an attribute that is expressed using a single value; ex., integer, floating point number, character; we also consider string to be primitive
2. Complex Attribute - an attribute that is expressed using multiple primitives; ex., 3D vector of floats
3. Compound Attribute – an attribute that is itself an Object containing other attributes; Animation has Keyframe attribute, Keyframe is an Object with its own attributes
4. Collection Attribute - an attribute that represents a data structure, such as a list, vector, or hash map. A Collection attribute can hold any other kind of attribute.

Given this language, we can express any visualization technique, behavior, or interaction style as a human and machine readable document capable of being interpreted at runtime. To demonstrate our concept for this machine portable language, we have created Bridgeworks Markup Language (BML). BML is defined using Extensible Schema Definition language (XSD). Documentation and source schemata are available online in the following locations:

Documentation of BML:

http://dev.bridgeborn.com/Bridgeworks/Schemata/doc/Bridgeworks.html

Schemata: (relative to http://dev.bridgeborn.com/Bridgeworks)
/Schemata/Bridgeworks/Bridgeworks.xsd
/Schemata/Common/Command/Command.xsd
/Schemata/Common/Directive/Directive.xsd
/Schemata/Common/Evaluator/Evaluator.xsd
/Schemata/Common/Node/Node.xsd



[Update 12.02.08: realized the schemata listed above do not contain all of the elements documented...not sure what happened...see BwSchools for full list. Schemata are best viewed in XMLSpy.]

We use Bridgeworks to provide our customers with the largest and most flexible set of visualization capabilities available to them on the market. We deliver these capabilities through a single lightweight runtime that scales between personal devices and large facilities. The runtime engine operates through simple input/output mechanism for parsing text-base, well-formed data, making it easy to integrate without a lot of API programming.

Platform Functional Requirements:

1. The capability must provide graph layouts (ex., radial tree, cone tree, force-directed graph)
2. The capability must provide chart layouts (ex., bar charts, line charts)
3. The capability must provide geospatial layouts (ex., maps, globe, terrains)
4. The capability must provide high-fidelity 3D modeling and animation
5. The capability must provide video and audio
6. The capability must provide hybridized views (ex., Extrude Map: geospatial + chart)
7. The capability must provide split views (ex., side-by-side comparison)
8. The capability must provide layered views
9. The capability must provide a temporal dimension
10. The capability must provide a mechanism for user-defined attributes
11. The capability must provide a mechanism for user-defined events, and triggers
    

Platform Technical Requirements:

1. The capability must provide multi-dimensional graphics (2D, 3D, 4D+)
2. The capability must provide an engine that can change views and behaviors at runtime using only a text based, interpreted language that does not require compilation in order to add/modify/delete content in real time
   
3. The capability must provide serialization and de-serialization of data and objects into and out of rich media views using:
   1. XML
   2. JSON
   3. CSV
4. The capability must provide serialization and de-serialization of standard object notations (SONs, ex., JSON) into and out of views
5. The capability must provide Object/Attribute Reflection
   
6. The capability must provide storage of domain data as Objects’ attributes loaded in memory
   
7. The capability must provide an extensible API
   
8. The capability must provide ECMA script language binding
   
9. The capability must provide COM language binding
   
10. The capability must provide Java language binding
    
11. The capability must be able to render from multiple 2D and 3D file formats, including but not limited to:
    
1. ESRI ArcView Shape
   
2. LightWave Object
   
3. Keyhole Markup Language (KML)
4. CAD Models (AutoDesk, SolidWorks)
   
5. JPEG
   
6. Bitmap
7. Extensible to other formats as needed


12. The capability must be able to render from data located in local and remote URLs
    
13. The capability must embeddable in the following software applications:
    
1. Internet Explorer versions 6 & 7
   
2. Microsoft Office 2003 Products: Word, Excel, PowerPoint
   
3. Mozilla Firefox
   
4. Win32 Graphical User Interfaces (GUIs)
   
5. .Net/C# GUIs
   
6. Java GUIs
   
14. The capability must provide an XML schema and a document object model (DOM)
    

Visualization as a Service

Visualization as a service is an extension of our core capability concept of visualization as a platform. Because we can express views, behaviors, and interaction models without deviating from our standards for reasonable portability we can publish and share views, behaviors, and interaction models just like any other content. This means that web services and other data servers can produce human and machine readable documents that describe visualization, behavior, and interaction. Furthermore, these documents can be parsed to create or modify any visualization, behavior, or interaction technique at runtime without intermediate compilation. Depending on transfer frequency, volume of data, and security constraints, data describing visualizations, behavior, and interaction can also be cached for use in “offline-mode” with so-called “rich Internet applications” (RIAs).


[Update 03.14.2009 - Just watched Tim Berners-Lee's TED Talk on Linked Open Data and recently watched Vint Cerf's Feb 09 lecture to FAA. Want to add here that a document-based approach is the best way to ensure that visualization follows the important lessons taught by these great teachers. Binary objects can be be linked using HTTP and metadata, but binary objects are not open. Binary objects also always require a specialized software reader to be interpreted. Documents, on the other hand, are what the Web was designed to link and text will always be readable by readily available software on the web. Even if one rendering engine disappears it is always possible for another to come along; ex., if rendering an old visualization document as graphics ever becomes important hundreds of years in the future. I realize I am contradicting this principle by encouraging the use of existing binary formats for images and 3D models, but this is a compromise I am willing to make at this point. Check out BML Hello World. KML is important and wonderful, but it is just scratching the surface.]

Tufte on document-based platforms

This just posted at the end of a new thread on Ask E.T. about Google Chrome:

(Emphasis added by me)

"How's the browser?

Thanks to the many contributors who pointed out Scott's kind link in the example of amazon search.

David Pogue's report in The New York Times suggests this browser leads to an integrated platform with non-proprietary formats (very good) but is still application-based rather than document-based (not very good)."

-- Edward Tufte, September 3, 2008

This is something I think constantly about - document-based platforms can serve end-to-end needs from data extraction, transformation, portability, and visualization. See also, the flow on Document Driven Visualization