Flow

Map of US Farmers' Markets

2011-02-07T22:09:00.000-08:00

NOTE: You may have to zoom in/out to get points to load. The map will only load 500 points at a time and is a bit quirky about paging data.

Gov 2.0 Events Calendar (Test)

2010-07-21T20:11:00.000-07:00

We OPML'd the DoD

2010-07-01T18:49:00.000-07:00

I'm just giddy about this. We OPML'd the DoD. Specifically, we published OPML for Joint Capability Areas. It turned out to be so easy, too. I wish I could easily convey the significance of this achievement.

JCA's are an exhaustive military taxonomy declaring hundreds of defined military capabilities. They are predominantly referenced in decision making for acquisitions and planning. All over the military there are organizations who are required to map every aspect of their existences to JCAs or else be denied resources. Consequently, nearly every “decision support” and “business intelligence” tool under the sun (there are hundreds, if not thousands of them; costing tens, if not hundreds, of millions) has a requirement for mapping JCAs. Just this week (in fact, after we decided to do this) one of our teams received a requirement to add JCA mappings in a database. The first question that came to my mind was, "how are they going to get that data into the system?"

You see, JCAs are published in formats that are only readable by people, and not in formats that can be used by tools. As a result, people repeatedly spend unnecessary time manually keying in JCAs just so they can be available for menu items, pick lists, and other input forms in their software applications. This is inefficient, error prone, and easily solved by publishing JCAs in Outline Processor Markup Language

So that's what we did.

Now system integrators can consume a URL-addressable, well-formed, hierarchical, text document as input to their application development. No more need to manually key stuff. No more typos. Point any developer worth his/her salt as this URL and they will be off to the races.

It's too bad the owner of JCAs doesn't publish more portable content, but maybe now they will.

Credit where it is due:

Our (2nd) wonderful co-op student from Rochester Institute of Technology, Ben Kaiser wrote the code. Longtime Bridgeborner, Rob Shell shared some SME-ness and was Johnny-on-the-Spot with this (mostly well-formed) HTML version of the JCAs. Thanks also to Sunlight Labs for feedback on choosing the right technology to get the job done. Raymond Yee provided an excellent model for us to follow (in response to this project idea from Clay Johnson).

Questions I Should Be Able to Ask My City (On the Web)

2010-06-27T21:01:00.000-07:00

It's too hard to ask a simple question about a city and get a simple answer.

This notion goes well beyond satisfying the need to know the answers. Our inability to ask basic questions about any city and provide simple answers is a much more profound shortcoming compared to just the answers themselves. Simple answers to basic questions are the foundational elements of more complex analyses.

When we rely on experts and representatives, elected or otherwise, to perform these analyses we expect accuracy and completeness. But how is this measured? No matter what the field of study, the answer has always been references, data sets, and peer review.

Today, especially in matters of public interest, the word "peer" is hard to define. All the more reason why satisfying the first two elements, references and data sets, is of paramount importance.

Cities should provide simple answers to basic questions.

Cities should provide simple answer to basic questions in a URL-addressable, well-formed, text format.

There are many mechanisms cities can use to satisfy this requirement; Web API, online spreadsheet, posted XML documents. Ideally, technology exists and can be integrated so that cities can answer questions via several of these mechanisms.

So what are the questions we would ask our cities, if we could? Following are a few to which I would like to know the answers. Moreover, given the answers to these questions in URL-addressable, well-formed, text format(s) the answers to these questions could make for some rather interesting, useful mash-ups.

Where is the location of each road and intersection under construction in my city?
How long has road/intersection X been under construction?
When will road/intersection X construction project be complete?
What is the list of each building construction permit in my city?
Where is the location of each building construction permit in my city?
What is the list of businesses that pay taxes in my city?
What is the list of businesses that pay taxes in my city, by type of business?
What is the list of how much each business paid in taxes in my city?
Where is the location of each business that pays taxes in my city?
What is the list of neighborhoods in my city?
How many houses are in each neighborhood in my city?
What is the list of schools in the city? (This should include private and church schools, too, if these schools are required to be registered with the city.)
What is the location of each school in my city?
What is the enrollment at each school in my city?
How many cars are registered in my city?
What is the list of most traveled streets in my city?
What is the list of traffic accidents in my city?
Where are the locations of each traffic accident in my city?
How many pounds of garbage are collected in my city?
How many pounds of garbage are collected in my city, by zipcode?
How many pounds of garbage are collected in my city, by neighborhood?
What is the list of events in my city?
When are events occurring in my city, by time period?
Where are events occurring in my city?
What is the list of conditional use permits in my city?
What is the list of all emergency response stations (fire, police, medical) in my city?
Where are the locations of each emergency response station in my city?
What is the number of emergency calls in my city, by time period?
What is the list of stations responding to emergency calls, by time period?
What is the frequency of emergency calls by station and time period?

[To be continued...]

Without a doubt, this list is far from exhaustive. Crime data is one of the most popular and reported-on data sets. I tried to come up with relevant questions for which data are not likely accessible. Certainly, schema are required for declaring the answers to these questions in our required format. Probably every question does not have to be anticipated a priori. We should also not assume that a particular question implies a specific data set. We often can easily get the answer of, for example, the location of each crime. Rarely, however, can we get aggregate data, useful in trend analyses. Perhaps what we need are structured data sets and Wolfram Alpha. In any case, a foundational platform of data is required. In any case, we should be able to formulate a basic question and get a simple answer (on the Web).

Given the simplicity of what is being asked, however, this should not be complicated.

Transparency and Visualization of City Data

2010-06-27T14:16:00.000-07:00

Last year my city was considering closing one or two schools (HamptonRoads.com, WAVY.com). One of the justifications cited was decline in student enrollment. Though I can no longer find it anywhere online, one of the enrollment artifacts put forth was this image of a column chart from a (presumably) Excel spreadsheet.

From Flow

To look at this picture (of data) one would assume there was a HUGE drop in enrollment, something in the vicinity of 50-60%.

But a closer look at the values reveals quite a different quantitative result. In fact, the actual decline from the peak in 1994-95 to the valley in 2008-09 appears to be 5,600 students. Keep in mind, I'm doing my best to derive the actual number based on the Y-axis scale on the picture. The data weren't provided. To make matters more difficult, the column chart is shown in 3D. This undermines the utility of the visualization by making it hard to determine an actual value on the chart.

The actual difference is 6,824 students, from the high in 1994-95 to the projected low in 2012-13.

6,824 students is 18.1% of the peak (6,824/37,707). While significant, 18.1% is a far cry from what appears to be a 50-60% decline.

Fortunately, when I requested the data they were provided to me.

This is valuable for two reasons. First and foremost, I get the data, not an interpretation of the

data. Second, and more the point of this post, I can show what this chart should look like and what else the data have to say.

Notice the slope of the decline and the differences between yearly observations. Neither are so exaggerated as the first image.

The differences in view are the aspect ratio of the graph and the value of the Y-axis origin. In the first chart the aspect ratio of the picture is nearly square, or 1:1. The starting value of the Y-axis is 28000 (an arbitrary value not found anywhere in the data). In the second chart the Y-axis begins at zero (0) and the aspect ratio is closer to 3:1 (which is probably too wide).

So then, which image and which set of chart variables is correct?

The answer depends on the basis for the query...or what you are trying to say. Actually the answer has nothing to do with "what you are trying to say" since you/we shouldn't be trying to say anything except to represent quantitative values in visual form. The right thing is to understand the nature of the analysis and the decision trying to be made.

In this case, we can assume with confidence that the Y-axis has no business starting at any value other than zero. In this case what is being claimed, via what is shown, is a long term (since 1990) decline, not a year-to-year fluctuation. Over the 14 year span the decline is obviously not in the range of 50-60%, therefore the chart should no imply so. Otherwise, if year-to-year deltas are of interest it probably does not make sense to show a 14 year span.

By the way, the data for recent yearly decline are -216, -200, +30, -17 between the years 2006-2010. 200 students city wide is hardly justification for closing a school. (Our school has over 600 students.)

In the end, my city chose not to close a school. I think that was a wise choice. But I have to wonder what was going on in the minds of the decision makers. What analyses were they using? Did they, too, realize the truth in these numbers or was it some other (political) factor. Did we just get lucky?

When cities make claims based on data they should publish the data. When cities, or anyone, shows you a picture of data you should question the picture and ~~demand~~ politely request a reference to the data. At minimum, know what you are seeing.

Mashing up Hampton Roads School System Data

2010-06-24T15:17:00.000-07:00

I'm tempted to not provide a narrative for this. The pictures really tell the story.

I was reading an article from my local "paper," The Virginian-Pilot, online. When I finished the article I noticed the "Quick Links" just below. To my pleasant surprise, the first category of links was "Databases." Truth be told, I'm not that surprised to find data at The Pilot, but what unfolded when I followed that link was far from the norm when it comes to online (local) journalism and data.

I clicked the link to "School Violence" and found the following page. If you look closely you will see there is a form:

Clicking "Search" took me to this next page. I was not surprised to see that the results were available in HTML only:

Whenever I come across data like this the first thing I want to know is whether or not it can actually be used as data. In order to be used/usable as data the contents of this HTML table need to be, at minimum, copy-and-paste-able into a spreadsheet.

Lo and behold... In about 20 minutes I had pasted all of the contents of the database into a Google spreadsheet:

From there I was able to create a few transformations of the data and about a half-dozen meaningful charts like this:

What if more news organizations published databases? What if journalists had a habit of digging into those databases with with these simple tools? Would we create better news products? Would we become more accurately and insightfully informed as citizens? Would our governments, businesses, and civic organizations make wiser strategy and better decisions? And if we think that the answer to any of these questions is yes, how do we sustain the "model?"

Transparency and the Digital Divide

2010-03-29T04:49:00.000-07:00

As I start this post I'm on the Orange line of the Metro heading home from Transparency Camp 2010. I timed my arrival almost exactly with that of the train using an iPhone app. Now I'm typing on a super-powerful laptop with a huge display. Many Metro stations have 3G access and even though I don't tether my phone to my computer to use 3G on my laptop, I'm sure it can be done. I have nearly all of the comforts of the digital age at my disposal nearly all of the time.

I often use these tools to stay in touch with what is happening in and around my community, my local and state governments, and the federal government. I use access to Web sites, data, and social networks to stay informed and engaged. These tools offer me many choices of how much info I want to consume and how much I feel like engaging.

But what can people who don't have these resources do to be informed and participate? What choices do they have for receiving information and offering feedback? So much of Gov 2.0 and open government relates to the Web that we must be careful not to exclude those who lack digital resources.

This topic came up in multiple sessions at Transparency Camp and we generated some good ideas (we think) on how to address this issue responsibly. One idea in particular that resonated was placing LED message boards around town to broadcast key indicators, initiatives, and citizen feedback.

The basic idea is to place LED signs at heavily trafficked locations. These signs would display information about the city in which people are most interested. Some of this content would be generated the city and some of the content would be generated by citizens. Information, especially that generated by citizens, would vary somewhat by neighborhood.

On the participation side of the conversation the minimum barrier to entry would be a mobile phone. Anyone who can send a text message can contribute. Of course, there would be other ways to contribute. On the information side of the equation would be these message boards.

This would be a way for people to simultaneously tell their city what they want and to see what the city is getting by way of feedback.

There is plenty of reason to think this can be effective, too.

But before I go on, I need to add some context. This all came up during a session I facilitated called "Local Government Transparency." In the beginning of the session I gave several examples of transparency happening at the local level. One example was Localocracy. Hart Rossman quickly pointed out that, while I'm able to easily cite these examples, most people hardly know what I'm even talking about. The point being: we need to make this relevant to the vast majority of people who aren't aware of what's going on in this space.

Keying on that, Bryan Sivak, CTO of Washington, D.C., shared with us his concerns about the "digital divide" in his city. In the poorest wards in D.C., broadband Internet access is about 30%. The point being: we can make all the cool open data and participation websites we want, but we won't be serving key constituencies in his city if those people can't access these online resources.

After some back and forth a gentlemen, whose name regrettably escapes me at the moment made a reference that reminded me very much of "The Blackboard Blogger of Monrovia," Alfred Sirleaf. Alfred is not just dealing with a digital divide. He has customers who are not literate. Still, he manages to provide useful information about what matters around them.

At lesser extremes we see other examples of this idea already working. Metro transit signs are one great example.

What else do people want to know about their city that can be easily provided on on a billboard in the public square?

It's worth noting that we don't necessarily need government to provide (and maintain) the billboard. Alfred Sirleaf makes a living doing what he does. People who visit his blackboard buy goods from him. We probably need government to issue permits for signs in public spaces. Then again, What if food vendors in D.C. hung LEDs on their carts?

We do need from government data that is open and accessible so that it can be easily used by anyone at low or no cost. Given this and a touch of inspiration from Alfred Sirleaf, maybe we can succeed at engaging citizens in their own government in ways that are widely accessible.

Post Script:
Gwynne Kostin pointed out the next day, that the digital divide is as much cultural as it is economical. There are plenty of people who aren't resource constrained who are nonetheless struggling with relevance of transparency and open government in their lives.

Thank you to Erik Hersman for the image of Alfred Sirleaf

How to Mash-up Open Data APIs + Google Docs to Visualize City Data

2010-02-23T08:59:00.000-08:00

Full size: http://www.screencast.com/users/kmcurry/folders/Default/media/342827dd-99f2-4744-9bf0-3eb3eec7e67c

What's Happened Since City Camp?

2010-02-06T07:32:00.000-08:00

At the start of the second day of City Camp we challenged ourselves to think about what happens after City Camp. We focused our sessions around concrete actions we would take after we all scattered back to our homes. Here's a quick rundown of some of our post-City Camp activities:

We've been editing the Open Muni wiki
We started a help-desk for journalists who want to use data
We're planning spin-offs and meet-ups in a number of different cities
We connected experts in crime data mashup & visualization with a local paper's crime data API
We stay active online
We started Colorado Smart Communities
We reported back to local government
Update 02.15.10: We set up a development environment for open source government data feeds

In fact, in their report back to City of Edmonton, Devin Serink & Ashley Casovan, listed six "next steps" for their city:

[Participate in] Inter-city collaboration
[Contribute to] Open Data Cookbook
Experiment in 3D virtual worlds
Develop understanding and excitement internally
[Initiate] Apps for Edmonton
[Support creation of] Code for Canada

To be sure, many of these activities were initiated prior to and/or separately from City Camp. That said, we all connected through City Camp to these activities and those connections are moving us foward.

P.S. This City Camper won an award.

Journalism, Data, Visualization

2010-01-30T21:02:00.001-08:00

This is time series data of crimes reported in Virginia Beach from 2007-2009. There are stories in this data. Perhaps some are significant. Perhaps many are not.

We don't know because we don't consider the data in this way.

What is that spike in 2007? If nothing else it is statistically unusual. But why?

What is the story behind the 247 crimes that were reported on this single day, June 1, 2007?

It doesn't look like there was a trending increase in reported crime.

Still, what accounts for the difference between the low of 78 crimes on April 20, 2007? Should we just chalk it up to summer tourism?

Is there nothing we can learn from this? Are there stories that might inform future generations?

My Initial Takeaways from City Camp

2010-01-30T07:29:00.000-08:00

I posted this in a comment thread in GovLoop; copying here so it can be read without requiring a login:

My biggest takeaways:

It's essential to get a good mix of perspective. We had civil servants, vendors, journalists, non-profits, and citizens. It would not have been as successful if it was gov-to-gov, vendor-to-vendor, or even gov-to-vendor.

It's essential that the conversation not revolve entirely around tech and data. In 2010 we can assume that technology and data are involved. We're just scratching the surface on process. And the processes involved are not just about methods and means for collecting-publishing-visualizing data. Providing greater opportunities to get citizens' voices heard and to increase their engagement in civic duty is important.

There is a new and important role for journalism: tell the stories behind the tech and the data. However, journalists may not have the education and knowledge to do this well. Interpreting stats is hard. I am excited to see Global Integrity stepping up to start a "help desk" specifically to work this problem. I think there is a new "extreme programming" model that papers could adopt; or perhaps to put it in terms papers already understand: pair your journalists up with data-viz-stats people like you pair them up with photographers.

People want what City Camp provided. We are going to learn from it, refine it, and keep it going.

Don't wait for me or Jen to keep City Camp going. Anyone can do this anywhere at anytime. Copy what works. Adapt for your local perspective. Just do it.

http://citycamp.eventbrite.com

2010-01-04T20:37:00.000-08:00

Visits to

http://citycamp.eventbrite.com

as of January 4, 2010

World

U.S.

Canada

Army's milBlog Runs on Word Press

2009-12-07T19:43:00.000-08:00

Word Press is awesome.

Iconography Service

2009-11-23T20:29:00.000-08:00

A Web Service that returns a MILSTD 2525C icon in PNG format given echelon (size) and proponent type (unit):

For example:

http://dev.bridgeborn.com/symbology/

SymbologyHandler.ashx?unit=I&size=2

This could easily be extended to include other iconography standards.

Code by @shadyproject

Wiktionary

2009-11-14T07:34:00.000-08:00

Update: 12.03.09 - It seems thefreedictionary.com is already providing most of what I'd like to see from a service: anyone can add (don't even have to auth), domain attributes, links to references, direct URL to entry: http://acronyms.thefreedictionary.com/ There are some obvious difference in approach. I want to go further with URLs. I'd like to be able to point to a unique defintion via a path, ex., /wiktionary/FOO/1. I also want to do lookups in other dimensions, ex., return all terms from domain:X.

The government is awash with acronyms. New acronyms are created daily. Acronyms create a barrier to understanding if they cannot be easily resolved, where easy = universal and universal = URL. There are many online dictionaries with entries that are found in Web searches. However, these return results only in highly formatted, not-well-formed HTML that is not always accessible through simple URLs. Furthermore, these dictionaries provide no way for the community to create and share new entries as they are needed. A simple solution to this is:

Use the cloud to store terms and definitions
Use Web services to return definitions through URLs as XML, JSON, and XHTML
Provide a simple form that lets registered users add and edit terms

We have created a proof of concept here using XAMPP and AWS:
http://dev.os.bridgeborn.com/wiktionary/

It's not perfect or even complete. For example, the XHTML returns errors from the W3C validator. But I think this is a solid start and I'd like to open it up and see it go further. To that end, I submitted it as an idea for Sunlight Labs. We'll see if it garners any votes of interest...

Bad Practices of Linked Data

2009-11-07T13:45:00.000-08:00

Urban EcoMap San Francisco is a great site that lets you explore emissions by zip code on a color-coded map:

Smartly, they also provide a "link" to download the data as comma-separated values.

Sadly, however, the link it not accessible!

Where is the URL for this data? It's hidden behind a Flash control.

Mashing up Virginia Beach Police Data with Google Docs

2009-10-25T08:30:00.000-07:00

Last Sunday, 10/18/2009, the editor of the neighborhood section of my local paper, The Virginian-Pilot, said that he was looking for ways to improve how he reports crime data. At the same time, I know that neighborhood crime is a hot topic in our community league meetings, casual get-togethers, and email lists.

Witness: hyperlocal supply and demand in Gov 2.0.

In fact, there's plenty my local paper can do to improve the reporting of [crime] data in our city. In the printed edition of the neighborhood section they only print the police "blotter" from the previous week. Online they use Google maps to report crimes by location, but interfaces are constrained and inconsistent.

What my paper should do to improve crime reporting is go to the source.

The Virginia Beach Police Department publishes crime data on a city website. Here's the URL:

http://www.vbgov.com/vgn.aspx?vgnextoid=8f770d066c0dd010VgnVCM1000006310640aRCRD&vgnextchannel=

6a0a1c17717ec110VgnVCM100000190c640aRCRD&vgnextfmt=default

Gnarly.

You can query for all crimes going back to January 1, 2006 (why only that far back?) up to the present day. But you can look at exactly 15 results at a time.

So what my local paper should do is appeal to the city to make [crime] data more accessible to everyone; where everyone includes The Virginian-Pilot. Short of that, the Pilot should take the energy it invests in supporting its Web presence and use it to scrape and publish city data. If only there were a "Machine Friendly" link next to the "Printer Friendly" link. (Both are designed to help people.)

Shortcomings notwithstanding, here's what Pilot reporters can do right now for free:

Compile and publish Google spreadsheets with charts:

http://spreadsheets.google.com/ccc?key=0AtBw9czGJjSTdGZGVzh3MzdzWEQ1SzJ6NTJDbl9taHc

http://spreadsheets.google.com/ccc?key=0AtBw9czGJjSTdFdlaEEwT2Y1RGotdGNWU2w4WDh3Nnc

Wow, who knew that ~~72% of all crime in Virginia Beach>~~ occurs at the Oceanfront?

Update: HUGE, HUGE ERROR in my reporting of this stat. The pie chart above does not show all crime in Virginia Beach. It shows only the top 5 neighborhoods.

This chart, below, is the correct chart. It shows that, while still a standout piece of the pie, 4% of all crime happens at the Oceanfront.

I'm feeling really dense right now, but in a way I'm glad this happened. This is a perfect illustration of a fundamental in data viz (and reporting).

Ten Things My City Can Do to Improve Our Website

2009-10-10T12:20:00.000-07:00

Publish events in iCal format
Publish electronic police reports as XML
Publish 311 data (as XML)
Geocode public works projects
Implement short, guessable URLs
Use a free mapping service
Update the Transportation Data Management System
Add a "Web-Friendly" link next to "Printer-Friendly"
Create a data catalog
Get a .gov address

Details to be added here...

Gov 2.0 Doers

2009-09-30T12:41:00.001-07:00

There has been some criticism that Gov 2.0 is:

Just another buzz word
Just people talking about stuff, no one doing stuff

I disagree, certainly with the 2nd point. Most of the Gov 2.0 people I know and/or follow on Twitter are the doers. Here's a list of who some of them are, certainly not comprehensive, in no particular order:

Brian Sobel, InnovationGeo, Are You Safe, iLive.at
Dmitry Kachaev, D.C. OCTO R&D
Philip Ashlock, Open Planning Project, Open 311
Josh Tauberer, GovTrack.us
Jim Gilliam, act.ly, GovLuv.org, TweetProgress.us, WhiteHouse2.org, and NationBuilder.com
Andrew Turner, GeoCommons, FortiusOne
Everyone at Sunlight Labs
Carl Malamud, public.resource.org
Jen Pahlka, Code for America
Leonard Lin, Code for America
Steve Ressler, GovLoop
Noel Hidalgo, New York State Office of the CIO
Raymond Yee, UC Berkeley
Peter Corbett, iStrategy Labs, Apps for Democracy
Kevin Merritt, Socrata.com
Hillary Hartley, NICUSA, Citizen Space
Guy Martin, Forge.mil
Silona Bonewald, League of Technical Voters, citability.org
Kevin Connor, LittleSis.org
Greg Elin, United Cerebral Palsy
Noel Dickover, DoD Office of the CIO, DoDTechipedia.mil
Jon Udell, Microsoft, ElmCity
Kim Patrick Kobza, Neighborhood America
Jay Nath, City of San Francisco
Wayne Moses Burke, Open Forum Foundation
Micah Sifry, Personal Democracy Forum
George Thomas, GSA
Alan Silberberg, You2Gov.org
Steve Lunceford, GovTwit.com
Joseph Porcelli, Neighbors for Neighbors
Luke Fretwell, GovFresh
Chris Rasmussen, Intellipedia, NGO
Pam Broviak, MuniGov 2.0
Bill Greeves, MuniGov 2.0, Roanoke County, VA
Jeff Levy, EPA Web Manager, Federal Web Managers Council
Adrian Holovaty, Everyblock, chicagocrime.org

BTW, we need both "talkers" and "doers." Some people are both. Some people are connectors. Not everyone is an implementer. I tried to stick to implementers; to pick people who have projects and/or organizations dedicated to or related to Gov 2.0. There are some obvious names not on this list. That's probably because they are not implementers. Doesn't mean they're not important. If I missed someone you think should be on the list leave a comment. Be sure to tell me what project they're on. I certainly don't know everyone.

I'll try to come back with links and pictures.

Views, Transformations, and Sources

2009-08-27T18:58:00.001-07:00

Visualizations, like any research product, should contain bibliographic hyperlinks to sources and transformations. (Click to englarge.)

Why Gov 2.0 Matters To Me

2009-08-25T09:51:00.000-07:00

I posted this here last night but took it down because it felt too personal and too much about my story. The reason I wrote it this way is because I didn't want to presume why it should matter to other people, but rather describe why it matters to me and let others find commonality in it...or not. Tim O'Reilly just posed the question: "What does Gov 2.0 mean to me?" I think he may be asking a slightly different question (definition vs. motivation), but given that I decided to put this back out there.

Friends and family have been asking me why I'm into Gov 2.0 and what I get out of it. I'm going to attempt to answer these questions for them and for me here in this post.

For me, Gov 2.0 is personal. I can't answer the question without relating it to my own story:

I was born on October 14, 1970 at Portsmouth Naval Hospital in Portsmouth, Virginia. My father was stationed at Naval Air Station Oceana in Virginia Beach. From where I sit writing this, Oceana is not more than a quarter of a mile away (as the jets/crows fly).

My mother was a teacher in the Virginia Beach Public Schools system. I grew up in that school system and my daughter is in it now, 5th grade. I have not always lived here. I spent about 10 years in the New River Valley of southwest Virginia for college, work, and graduate school. I did not intend to come back to Virginia Beach after grad school. In 1999 I was graduating with a Master of Science in Computer Science and Applications degree from Virginia Tech. It was decent ticket to anywhere and I was looking west to San Antonio, San Francisco, and Redmond. But events unfolded and I transitioned from a part-time graduate research assistant at Virginia Tech to a full-time Research Associate at Old Dominion University Research Foundation, in Norfolk, VA. Fast-forward to today, in the late summer of 2009, and I'm right back where I started. I think my internal magnets have this lat/lon set as my home base.

I'm not at ODU anymore. Today I'm a government contractor at a company I co-founded. Mostly I am a Defense contractor. I didn't set out to be that either. In fact, I was explicitly trying to avoid it. I don't want to get into all of the reasons why. It's far too complicated for anyone but me to understand and I'm not even sure I do. At risk of abusing a metaphor, suffice it to say that my internal magnets draw me to other things. But I have grown and learned as Defense contractor. I have banished narrow prejudices and have adopted new world views. I have also rekindled something in me that was instilled at a very early age by my parents: the importance of public service.

I can still see my dad standing in our driveway at my childhood home, as I would have looked up to him, telling me about public service. I remember he tended to talk specifically about civil service. Though he was in the Navy he didn't necessarily talk about military service. Grandma Curry worked for the U.S. Customs Service and I know my dad admired that. Dad also talked a lot about how involved Grandpa Curry was with Miami-Dade Parks & Recreation, in Miami, FL.

I've rarely shied away from taking initiative where I see something needs to be done and participation in organizations comes naturally to me. I've ignored the call to service once or twice. After 17 years growing up a Navy brat, the effects of having a parent coming and going every 6-8 months, and lacking any of my own life direction, I steered away from ROTC and a long term commitment to a military career. I could have gone into civil service, I suppose, but another thing about my internal composition: I'm an entrepreneur and I don't care much for layered, abstract bureaucracy. I'm not so much into rising up through the ranks as making my own way. It's not that rising up through the ranks is bad. It's highly admirable, in fact. It's just not me. For me, patience is a trained virtue. If there's one thing I've learned as a government contractor is that working either with or for government requires intense patience.

So here I am. I'm a Generation Xer. I have not served in the military or civil service. I am easily frustrated by government process. Yet, I have an intense inclination toward public service. How do those add up? What can I do to serve government? (I think this is a question lots of people like me are asking themselves.)

I think today the answers are non-traditional.

I am active in the PTA for my daughter's school. I try to remain active with our Community League. For the past two years I've organized our neighborhood's participation in Clean the Bay Day. Before then our neighborhood didn't participate in CTBD. Of course, these aren't government activities. I think they help government activities, though, and that's significant.

For me, Gov 2.0 represents a near-perfect fit with my personality, direction, and goals. I feel like it was made just for me or that I was made just to live at this time. I really do.

There is and will continue to be no shortage of debate about what Gov 2.0 is, exactly. In fact, during the second week of September there is a "summit" in Washington D.C. dedicated to the topic. Is it tech? Does it include non tech? How does it work? Who can participate? I'm not sure framing it is as important as learning to recognize it from multiple perspectives and acting. I tend to take a literal interpretation. For me, that means there is a distinct connection between Gov 2.0 and Web 2.0. Therefore, for me, Gov 2.0 tends to be significantly Web/tech enabled. But as someone experienced in working for government, I know that tech requires policy and that both are intended to work for people.

This year I attended three important "unconferences" for Gov 2.0 movement: Transparency Camp East, Gov 2.0 Camp East, and Transparency Camp West. I will be at the Gov 2.0 Summit in a couple of weeks and at Gov 2.0 Expo next Spring. What most interests me is how I can use Gov 2.0 techniques to improve the way my city functions. I am interested also in how my customers will respond to Gov 2.0 and how I can apply it to their needs, but my motive is not winning more contracts. Sure, some will say that if they think it's what other people want to hear. Let there be no doubt that I want my business to succeed and I want to prosper from it. But I'm willing to take it on faith that both things will happen if I just keep following my passion and don't resist the pull of my internal magnets. I just want to "work on stuff that matters."

Increasingly, I want to work on stuff that matters here in Virginia Beach and in my region. But I also want to work on stuff that matters for my state, our nation, and our world. I am particularly drawn toward applications in education, where I see a growing gap between public and private education that is not just affecting the fringes, but is squarely squeezing out the middle and making our nation "dumber" as a whole. I see Gov 2.0 as a means for correcting this troublesome situation. (In general, I think Gov 2.0 can be particularly effective in areas that lack models, process, and funding.)

Early next year I hope to be kicking off the inaugural CityCamp, along with Jen Pahlka from Tech Web. Our goal is to start an unconference around the theme "Gov 2.0 goes local." Jen also has another project in the works that is very much in line with this theme. It's called Code for America and it's modeled after the successful Teach for America program. I look forward to seeing that take shape.

So, this is about the examples set by my parents (and their parents) and the lessons they taught me. It's about wanting to be a part of the solution in my community and my school system; not just a sideline complainer. It's about recognizing that I have something to offer that government needs. This is my opportunity to serve. This is why Gov 2.0 matters to me.

U.S. Code as Linked Data

2009-08-14T18:00:00.000-07:00

Recently the Office of the Law Revision Council released a request for proposal (RFP) to upgrade their website.

"The Office of the Law Revision Counsel prepares and publishes the United States Code, which is a consolidation and codification by subject matter of the general and permanent laws of the United States."

We responded to that RFP. We = Bridgeborn, Inc, and Business Bullpen, LLC.

The Law Revision Council's recognition that the online home of the United States Code needs upgrading translates into a wonderful opportunity for the LRC, our companies, and the American People. This is much more than an opportunity to redesign web pages for an online presence.

This is an opportunity to publish the U.S. Code as linked data.

Linked data is important for the U.S. Code because it will make the Code more searchable, navigable, and usable by orders of magnitude. Linked data will also increase accessibility and lower costs of integration by making it easier for more consumers to treat the information according to their needs and possible constraints.

Sites exist already to provide the U.S. Code through styled web pages. The Legal Information Institute of the Cornell University Law School, for example, publishes a searchable HTML index of the U.S. Code. This version, however, is not well-formed, linked data. These sites also omit important text included in the official record published by the LRC, such as the Positive Law Codification actions that have been taken. These sites play an important roll is the dissemination of U.S. Code, so it is our hope that this effort will also make the U.S. Code more accessible and usable for consumers like Cornell's LII.

The LRC Web site isn't too bad really. Essentially what it needs is a global site navigation scheme, search on every page, and a good Cascading Style Sheet. There are features that could be added, such as public request for comments with voting. But the most important thing anyone can do with this site is Tag the

Code.

For example, the following text:

-CITE-

THE ORGANIC LAWS OF THE UNITED STATES OF AMERICA

-End-

-CITE-

USC THE DECLARATION OF INDEPENDENCE - 1776 01/03/2007

-EXPCITE-

THE DECLARATION OF INDEPENDENCE - 1776

-HEAD-

THE DECLARATION OF INDEPENDENCE - 1776

-MISC1-

THE DECLARATION OF INDEPENDENCE - 1776 (!1) IN CONGRESS, JULY 4, 1776 THE UNANIMOUS DECLARATION OF THE THIRTEEN UNITED STATES OF AMERICA

...

Could become:

From here, there is nowhere we can't go with the Code. We can put put it in any container, we can transform it into any view, we can access it from any device. Given tagged, well-formed, linked data, we can address every element of U.S. Code from a standard Internet URL.

While tagging the code with XML may not fully constitute linked data, it is a big step in the right direction. Decorating the those tags with RDF is easily accomplished.

There is no point in enumerating the potential applications of an endeavor such as this. They are infinite.

Visualizing Army with Flare

2009-07-29T21:16:00.000-07:00

The Mr. Miyagi School of Software Engineering

2009-06-12T08:38:00.000-07:00

This is a re-post from a journal entry I originally wrote over on Slashdot in 2004. I've made some minor changes and updates for this entry.

Background: This was an concept born out of necessity about 5 years ago. I needed a way to train someone with very little experience to work with Bridgeworks but I couldn't afford to spend hours and hours of my own time working with him; too many of my own responsibilities and deliverables. It was very successful and now he is our best Bridgeworks app developer. Fast-forward to today when I'm interviewing CS students from a local university for a summer internship. This is a (well) paid position for which I need someone with decent coding skills who is a self-starter. Unfortunately none of my candidates have any practical skills whatsoever. (I don't know what the universities are teaching, but it's not what we need in industry.) With these candidates it would be like starting from square one. Obviously I'm not going to pay what I'm willing to pay for people who don't have the quals. I am willing to train such a person as an unpaid intern, but that is not without its catches. I still don't have the time to spend hours and hours with these folks. Furthermore, ethically, unpaid internships are intended for the sole benefit of the student. My company should not benefit in any way from the labor of unpaid interns. Practically speaking, though, you can't train someone without giving them tasks. Re-enter the Mr. Miyagi School of Software Engineering.

Lessons on how to train a junior programmer:

Let's call him Danielsan.

Mr. Miyagi was a very wise and clever sensei. His methodology, loosely translated, is perfect for any small software company that is bringing new developers into the system. The reason Mr. Miyagi's method works so well is because it provides intense, immersive exposure to the most important lessons while demanding relatively few additional resources from the instructor(s). Think about it. While Danielsan was busy painting the fence and sanding the deck, Mr. Miyagi was out having the time of his life!

The length of each lesson is to be determined on a case by case basis.

Lesson 1: Write SDK Documentation

Even the best developers can be notorious for not adequately commenting their code. Good documentation of an software includes both programmer's notes and comments for automated documentation (e.g., doxygen). This oft neglected task is perfect for Danielsan. An excellent way for him to learn the software from a developer's perspective is to write the documentation that explains how it all works.

Listen carefully when Danielsan asks questions about the existing code base. Discourage him from asking too many questions, except regarding complex concepts. It is important that Danielsan develop his own understanding of algorithms, relationships, dependencies, etc.

Lesson 2: Build Company Demos

Documenting the code shows Danielsan how the software developer sees things. Danielsan also needs to see the software from users' perspectives. "Users" include programmers who develop applications from the software and end users of the applications that are developed. Ideally, these would be two separate lessons. Knowing that time and money are always issues, these lessons can be condensed into one by having Danielsan build the company demos.

Architects and senior engineers loathe building company demos. While they are often happy to write test apps for in-house use, company demos bring with them a mountain of maintenance headaches and customer support issues. Whether or not your company is big enough to have it's own department(s) for maintenance and support, it's worth putting Danielsan to work on company demos so he can get his hands a little dirty and see first hand the challenges facing maintenance and support team(s).

Lesson 3: Clean House

Many companies have coding standards that must be followed by all code writers. These standards help everyone to write clean, consistent code that everyone can understand. Unless you work at a sterlized laboratory, it's a safe bet that your house always needs cleaning. A great way for Danielsan to learn this important lesson and also develop habits that are consistent with the team is to set him to work checking for adherence to company coding standards, leaks, potential security issues and the like. It's also a convenient way for you to get your code checked by a fresh pair of eyes.

The Successful Sensei

The successful sensei will know that these lessons are not opportunities for him to relax his own standards or to set Danielsan to work unguided. The successful sensei practices what he preaches. He knows which lessons Danielson must learn on his own and which require guidance. Naturally, Danielsan may occasionally bemoan his instruction. Perhaps he will consider that his training is too rote or mundane. Since you can't just throw a bunch of karate maneuvers in his face to show him what he's learned, it's important to sometimes let Danielsan work on things he finds interesting and fun. Ask him to write stand-alone utility apps that your company might need. Give Danielsan isolated new tasks within the SDK, perhaps something that requires he work with others to design interface requirements, resource requirements, etc.

to be cont'd.

Axiom 1: Tooltips are better left on.

If you leave your tooltips on, chances are better that you will learn something new each time you use your application(s). For Danielsan, tooltips are especially useful when they instruct him about fundamental principles of programming, those that transcend applications.

Axiom 1a: Some tooltips are better than others.

On Ambient Visualization

2009-05-18T20:17:00.000-07:00

I want visualization to be less a part of a specific application that I go to and to be more of a natural extension to the computer itself, available from everywhere. I want visualization to an ambient experience. When I encounter any table of data in any document container I'd like to be able to quickly view it as a column chart without starting up a chart-making or data processing application, without shuffling around through copy & paste. I just want to select rows and columns an pop a "window" with a chart in it in one easy step. If I can recognize with my eyes that a table contains place names or lat/lon pairs then a computer ought to be able to map it with minimal intervention on the part of the user. I should also be able to put my selected, obviously geospatial data on a virtual Earth model. With just a little more imagination I can see turning lists and tables into nodes and edges, viewable with graph layouts. Think Enso for visualization. I might want to do more than just look at my chart, map/Earth model, and graph. I might want to start to interact with these views (assume independently for now). It starts to seem like I need an application to do that, but I'm not ready to jump the gun. This is still in the realm of a capability and not necessarily an application. Applications start to assume containers and domain-specific use cases. Most visualization techniques have standard, "off-the-shelf" things you can do with them given basic commands or input devices. Charts can be sorted and transformed into different layouts. One can pan, zoom, and rotate maps and terrain. The technique of "drill-down" and "roll-up", which can be applied to any visualization technique, is nothing more than navigation of linked data at multiple levels of detail and sometimes across multiple view contexts. At what point do is a specialized application needed more than a capability? We may be overly conditioned to assume the application model when we think of software as having utility. This is changing rapidly on the web. (It was always thus on the Unix command line, yes?) Visualization ought to change with it. Leave the application building up to subject matter experts with an application domain, not to software programmers. Ah but wait, lest too much be read into a passing editorial remark. Obviously software programmers play a key role here. The tendency among programmers who attempt to answer that call is to build an "application building framework." Again, the assumption is that subject-matter experts always need an app to make use of viz. I wonder why visualization software shouldn't be a part of an operating system; a core capability for any application or purpose. There have been wonderful advances made through web browser extensions, but even here visualization is at best an after thought applied to a mostly universal application. (I say "mostly" b/c there are no less than 3 different web browsers installed on my one operating system.) What happens when someone emails me some data in a flat file that I open in a text editor? Instead what is need are document object models for visualization techniques and runtime software that can parse viz documents on the fly. The runtime is optimized for robust interaction and attribute manipulation of high level visual artifacts, not application-specific tasks. This runtime can be invoked from a background process or "embedded" (contained in, called from) an application runtime. Devices having different display and user interfaces can choose how to represent what are otherwise well understood visual metaphors. Data can be more easily passed around and visualized simply by passing text documents describing interactive, dynamically updatable (or not) views. (This seems inherently more secure, too.) Only this way is ambient visualization possible; something that is available everywhere on my computer, no matter what kind of computer/device/hardware platform I am using.

Carl Malamud for Public Printer of the United States

2009-03-09T13:58:00.000-07:00

I endorse Carl Malamud for Public Printer of the United States

(Administrator of the US GPO)

Enterprise Architectures As Data 2

2009-02-15T08:54:00.000-08:00

I wrote about this topic last December. Now I want to drill down on a key element of our approach for turning enterprise architecture models into queryable sources. Furthermore, I should address why anyone would ever care about "turning enterprise architecture...blah blah blah."

There is a technology aspect and a human aspect. Both are necessary, but technology alone is never sufficient. Far more important to our success has been a strategy focused on answering key stakeholder questions.

My experience is mostly with Department of Defense Architecture Framework (DoDAF), so I will use examples from that domain. But to the best of my ability I will use language that is portable to other domains.

So, what are "key stakeholder questions?" To what does "key" refer, "stakeholder" or "questions?" The answer to the second question is: both. The answer to the first question is: depends on which stakeholder(s) we are talking about.

In the Army, and DoD in general, organizations are broken down at the general officer level from 1-9 as follows:

HR
Intel
Operations
Logistics/Sustainment
Planning/Doctrine
IT
Training
Finance
Civil-Military Cooperation

These are the key stakeholder perspectives at the highest level.

An Operator wants to know, "How will architecture help me win the fight (i.e., get the work done)?" A Budget Analyst wants to know, "How does architecture help me making smarter spending decisions?" These are reasonable questions.

UPDATE 2009-09-28: THERE IS AN ERROR IN THIS GRAPHIC. THE G8's perspective in this case is "Force Development." He is asking the question, "Is the Force we are developing in line with my capability expectations?"

Indeed, no model exists except to answer questions.

But when presented with architecture in forms that speak only to the IT/engineering perspective, i.e., schematics, flow diagrams and the like, the multi-stakeholder conclusion tends to be, "Architecture doesn't help answer my questions." That's unfortunate because architecture, especially at the scale and complexity of federal government, can be effective in answering many useful questions for all perspectives. The trick is to:

Understand what questions are being asked from each perspective
Extend architectures from models that produce schematics into queryable sources that can produce schematics and also answer questions
Extract the answers to those questions automatically to an open, text-based format
Transform answers into views that are "fit for purpose," i.e., can be understood and used from many perspectives

Therefore, the question IT people should be asking is, "What can I do to make architecture better able to answer stakeholder questions." To be sure, there are questions architectures can't answer. But it is more productive to focus on the questions that architecture can answer. Following are actual endpoints in use (though they don't point anywhere in this post):

GetCountOfPlatformsForArch
Returns an Xml Node containing a NewDataSet for the distribution of platforms for the given architecture
Answers the question: What are the platforms of this architecture and how many of each?

GetCountOfPlatformsForArchAndTOE
Returns an Xml Node containing a NewDataSet for the distribution of platforms for the given architecture and TOE
Answers the question: What are the platforms of this architecture and TOE and how many of each?

GetEquipmentForONN

Returns an Xml Node describing the equipment used by the given ONN in the given architecture
Answers the question: What equipment does this ONN use in this architecture?

Believe it or not, this type of view can be automatically produced

from an enterprise architecture model

There is added value in this approach, too. From the technology side of the coin, opening up architecture models through relational databases and ubiquitous, text-based formats makes it much easier to relate architecture data with other data. An architecture can tell you "how many," but "how much" is probably in another database. In fact, there are about 85 software programs the Army knows about that are being used to track costs for systems. To be sure some will be eliminated. But many will stay and those that do must support data portability.

Ok, so...why does any of this matter?

Let us start with the fact that government is an enormous compiler of data. It really doesn't matter if we are talking about enterprise architecture, a Line Item Number database, a "portfolio management" system, or even good, old-fashioned spreadsheet. Analysts will never part with their spreadsheets. Wonks stuff data into tools.

Unfortunately, many useful tools that are good at capturing data are lousy at both storing it and saving it back out. Applications too often co-mingle their own data with the subject matter data, making it difficult to separate later. Many tools don't have export in mind. (All tools should have export in mind.) Owners of online, enterprise databases typically only offer their contents in human-readable formats; PDF and HTML pages. Applications built for one perspective are lousy at supporting other perspectives. The list of interoperability challenges goes on.

A battalion commander is not typically nimble with UML. Same goes for a budget analyst and resource director. In the world of enterprise architecture the consequence is that a lot of people are taking a lot of time away from their primary tasks so that they can manually shuffle data into views their many customers demand. Time is money. My back of the napkin estimate is that architecture shops are working twice as hard for half the output that they could otherwise achieve if they offloaded the ETLV problem to IT and got back to building architectures. Of course, plenty will point out that architecture and reality are often different. That's a topic for another discussion. Like it or not, enterprise architecture plays a huge role in government organizations, especially DoD. This is just about getting better return on that investment. It's opportunity for both a current cost savings and future "force multiplier."

Finally, I would argue that the strategy of answering key stakeholder questions is useful to knowledge management in general, regardless of subject matter domain or technology form factor.

"Response" to Previous

2009-01-27T20:55:00.000-08:00

Update 12.28.09: I just had to record for posterity the flames from some folks who tagged my previous post in Reddit. (So much for polite, constructive discourse.) In truth, I would like to have this conversation with some of these folks. Even the harshest ones. But I don't want to have to register for a service I don't use and I don't want to get into flame wars over comment threads. I get the frustration. I've experienced it. I too have been and remain rather skeptical of the hype. Some level of hype gets things noticed and forces conversations that need to happen but aren't. Some of what I said was written and/or read poorly. For example, Mac/Office is about culture, not interoperability. I know that Google came before Web 2.0. That was expressed poorly. My company also was doing a lot of this stuff before we ever heard of Web 2.0. I can't speak for Tim O'Reilly, but I feel confident he realizes he didn't invent something. He made salient observations about the Internet and human behavior and how the two have, do, and perhaps ought to work together. Many of these concepts go back decades to the roots of computer science and the Internet. I've been in workshops with Senior Technical Fellows, having extremely well-qualified CVs, who said, "we proposed all this in the beginning." My response was, "Exactly. Isn't it past time we got back to that?"

And I know how radical ("insane") my idea sounds, but I am not saying government should fund Web 2.0. This is not about a bailout, as some seem to think. Some of these comments come from the position of having no idea how government runs. I'm saying government is actively spending money, money is being wasted, and I know from education and direct, relevant experience that several of the concepts articulated in the original essay can help tax paying citizens spend more wisely.

Consider this from the Army:

"The Army spends, under 85 programs, approximately $6.7B annually on Information Technology (I.T.) without a method to converge these systems into a centralized infrastructure designed to improve robustness and dynamically deliver web services to hundreds of thousands of users while reducing risk"

And that is just service within DoD, DoD just being one (the biggest) government spender on IT. It's full of waste and the reason isn't just because Web 2.0 is magic Koolaid or that we are already on to so-called Web 3.0. It is because the government still looks at the Web the way it used to be in the 90s; i.e., with a 1.0 mindset.

More concrete thoughts on this later on how exactly government might use Twitter, for example. Think SMS. Twitter is a metaphor. The folks who run Twitter - for example - just might be the Pros from Dover who can help save government from itself. Think FEMA and USAID...

Here's the juicy stuff:

Haha, this nutbag thinks that the government should fund web 2.0. And he's serious.(kevincurry.blogspot.com)

submitted 4 days ago by candlejac

Kitchenfire 3 points 4 days ago[-]

""The Bigs," i.e., large-cap companies that provide most of the contracting labor, are not at all oriented to innovate in the Web 2.0 technology space. You don't see Macs anywhere. You do see MS Office everywhere. "

I wonder if this guy knows that Macs can run MS Office. Or that he's insane.

turkourjurbs 3 points 4 days ago[-]

"Google does a good job selling into government with its enterprise appliance model, and with more than just search. But, of course, Google is a massive company."

SIGH!!! Google was aroung long before Tim O'Reilly decided he needed more money, and made up a completely inaccurate and bullshit term to describe something we already have. Even if he didn't decide to look like a technological ignoramus, we'd still have Twitter, Facebook, etc. without calling them something that makes absolutely no sense at all.

Please, point out which "Web 2.0 Server" I should be using and which "Web 2.0" browsers will work with them. Every underlying technology that's considered "web 2.0" is the same technology that's been behind the web since before there was Web 2.0 We have the web. We have web sites. There is nothing more to it.

Maybe if we somehow get rid of the head nutbag (O'Reilly), the rest of the web 2.0 delusionists like this one will go with him.

skymt0 2 points 4 days ago[-]

Please, point out which "Web 2.0 Server" I should be using

That would be lighttpd, according to their home page.

cochico 1 point 4 days ago[-]

pffft! We're already working on Web 3.0

funkah 1 point 4 days ago[-]

Crazy, sure, but I'm not exactly loving the idea of giving tons of money to banks who lost hundreds of billions because of shitty risk analysis, either.

candlejac 1 point 4 days ago[-]

At least their risk analysis is better than Twitter's.

funkah 2 points 4 days ago[-]

I don't even know what that means.

candlejac 1 point 4 days ago* [-]

Twitter is a company with no business plan. They seriously are closest to the underpants gnomes

Build website for 140 character microblog posts
Pay carriers to send txts with updates to subscribers, while not charging for this service
Pay to receive texts on a shortcode
???
Profit!

grilled_ch33z 1 point 4 days ago* [-]

I'd argue that it's more like:

Build website for 140 character microblog posts
Pay carriers to send txts with updates to subscribers, while not charging for this service
Pay to receive texts on a shortcode
???
???

edit: how do you do numbered lists?

candlejac 2 points 4 days ago[-]

numbers and periods, but you must not know much about the underpants gnomes

Ac3 2 points 4 days ago[-]

Well he clearly will not profit.

Q: How Does Web 2.0 Make Money? A: Government.

2008-12-28T20:32:00.000-08:00

A lot of folks are wondering how Twitter will monetize. Will they sell premium services to businesses that want to make Twitter part of a communications strategy? What about Flickr, YouTube, and Facebook? Are the ads working? Regardless, if this is Web 2.0 then why are we still talking about subscriptions and eyeballs on pages?!

I want to suggest another strategy: Sell to government.

I don't mean the general sense. I mean, specifically, Twitter, YouTube, and Facebook and a great many others should set up for-government operations. I wrote to the Delicious team years ago asking when they were coming out with a solution that I could bring to government, i.e., in-house. I never got an answer. Regardless, figuring out how to get these companies oriented toward government is not straightforward. Most Web 2.0 companies literally could not be further outside The Beltway. I suspect they don't have much in the way of strategy for state, tribal, or local either. Google does a good job selling into government with its enterprise appliance model, and with more than just search. But, of course, Google is a massive company.

Culture has a lot to do with things. The Pentagon is not a T-shirt and flip flops kind of environment. "The Bigs," i.e., large-cap companies that provide most of the contracting labor, are not at all oriented to innovate in the Web 2.0 technology space. You don't see Macs anywhere. You do see MS Office everywhere. I'm not entirely sure what conclusions can be drawn from these observations but I am sure the observations are significant.

I suppose the best example of Web 2.0 penetration into the government space is tele-presence. Adobe Breeze is ubiquitous on Defense Knowledge Online (DKO). Just about anyone with a DKO government account can create or attend a meeting. But tele-presence probably isn't the first thing that comes to people's minds when asked to name a Web 2.0 technology and I'm not sure Adobe is the best example of a Web 2.0 company.

Yes, there is something different about Twitter, Flickr, YouTube, Facebook, Delicious, et al. I happen to think that something different - whatever it is - translates into unrealized opportunity for both buyers and sellers in the government space. I choose to focus on these technologies specifically and there are others that I include. ProgrammableWeb has a solution for the registry problem, for example. I don't represent any of these companies, by the way. I call them out as (mostly) well-known examples of capabilities the government needs. I don't really care if Flickr, Picasa, or PhotoBucket is the image repository of choice. Vimeo and YouTube can and should compete for the video infrastructure.

The point is that government needs platform solutions.

Simple, content-based, platform solutions are the most obvious plays for Web 2.0 in government; images, video, audio. The federal government processes a staggering amount of this stuff. The DoD may be the first to get into the mix with TroopTube.

There also are plenty of outfits that would use social tagging tools if only they could bring them in house. By bring them in house, what I mean and recommend is providing enterprise services solutions. It's nice to have applications that provide tagging, but applications are seldom the best enterprise solutions and are hardly social (except, perhaps, in MOSS). Tagging is a domain, a Web 2.0 partition, if you will, unto itself. It is a simple-enough-but-not-too-simple utility that scales and it can be integrated with just about any other application, regardless of whether or not the application was designed with tagging in mind. Yes, government needs strategic guidance and support for tagging services and Web 2.0 tagging companies are just the ones to provide it...if we can figure out how.

The current providers, a.k.a. "The Bigs" are not oriented to provide Web 2.0 tech support.

This is either an opportunity to get new business before The Bigs or to create a business of showing them the way. In any case it is an opportunity to make money. More specifically, it is a way for Web 2.0 technologies to make money. In the government services space the biz-speak is generally referred to as "priming" and "subbing," as in: you are either a prime contractor or a subcontractor. I don't see many Web 2.0 companies subcontracting to big, traditional system integrators, though.

The government is not set up to acquire Web 2.0 technologies.

While services contracts are fairly common and understood, things are less well-defined in the products and solutions space unless the products and solutions are ubiquitous or otherwise extremely well known. Government, especially federal, wants to buy everything in bulk. See also: Indefinite Delivery, Indefinite Quantity (IDIQ). New technology is extraordinarily difficult to insert, especially in secure environments. Protocol, procedure, and process rule the land. Force-fitting into an existing model is too often the preferred method.

We are getting better at defining and avoiding undue process but process, by definition, is: 1) necessary and 2) inherently situational.

When doing business with the federal government, it's important to know how business is done inside The Beltway. There are most probably things in the mix that need to be undone, too. To a meaningful extent the situation is not different at state, tribal, and local levels. Education is needed on both sides of the buyer/seller relationship. We might "change the world" in the process of implementing Web 2.0 for government, but requisite is the obligation to have a fair understanding of "the world" first. The obligation goes both ways but mostly falls on the shoulders of sellers.

Choices for Web 2.0 companies to make money by doing business with government:

Become or spin-off an enterprise systems integrations unit
Sell enterprise systems solutions a) to government b) to system integrators (Note: probably can't sell solutions to government without either being an integrator or having the support of one.)
Consult a) to government b) to system integrators on enterprise systems

Of course, "partner" is an option, but still implies one or more the previously listed options.

[Update 9:47 PM - I've decided I really don't like these choices at all. Need to come up with an altogether new business model, perhaps...probably]

There's so much more to this than I can wrap my head around now, certainly more than I am prepared or qualified to comment here.

[Update 9:47 PM - Forgot to comment on need for and evidence of government investment in backbone instracture and understanding of cloud architecture; significant issues arise once a bunch of these services are running around on a single network. And as always, security is different and harder.]

I think it's time Web 2.0 companies, government, and large-cap contracting companies had a grand introduction to one another. Believe it or not, there are plenty of people who have never even heard of Web 2.0.

Common Web 2.0 Services for Government Brainstorm

2008-12-22T19:23:00.000-08:00

SMS - ex., Twitter
Expertise Location - ex., Facebook, LinkedIn, semantic profilers
Tagging - ex., Delicious, Magnolia, Stumble Upon
Image Sharing- ex., Flickr, Picasa, PhotoBucket
Video Sharing- ex., YouTube, Vimeo, TroopTube! (12.28.08)
Audio Sharing - ex., SoundCloud, HuffDuff
Document Sharing - ex., Google Docs, MOSS, Word Press, Blogger, Tumblr
Registry - ex., ACME.ProgrammableWeb, Wiki (?)
Tele-presence - ex., Breeze, WebEx, GoToMeeting
Search - ex. Google, Yahoo!
Visualization - ex., ManyEyes, Swivel
Dictionary - ex., Merriam-Webster.com (http://m-w.com/dictionary/[WORD])
Data Transformation- ex., ??
[Update 12.29.08] Geospatial, mapping - ex., Google Earth/Maps, Microsoft Live Virtual Earth/Maps

Utility model (i.e., like electric, water, natural gas)
Specialized content and application services infrastructure
Government needs a strategy for inserting these technologies
Government strategy must be capabilities-based and vendor indifferent, yet cannot be generic
How exactly/specifically can government do business with Facebook or LinkedIn, for example?
To what extent will/should third-party integrators be involved? Is Twitter likely to provide labor resources for technology insertion or would they just want to license the platform to third parties?
Companies listed above need a strategy for doing business with government. Some have them, but most don't.
The mutual strategy should be for companies to implement their architectures for these platform services in public and government-managed domains.
Probably the biggest hurdle is the massive amount of process and procedure required to navigate the government marketplace and interact with customers
Culture gaps

More later...

[Update 12.23.08]

Security model is much more complicated than just protecting access to personal information; ease of and tracking of information flow a greater concern
Security models need to be reconsidered by both buyers and sellers

A Pattern is as a Pattern Does

2008-12-13T18:27:00.000-08:00

I had a great conversation the other day with Mike McKinney. One topic in particular really stuck with me and now I can't seem to shake it. Mike and I had been talking about design philosophies and what it takes to do smart implementation, especially as a government contractor. We both lamented the environment of design and implementation bloat in which we seemed to find ourselves. Bloat does not fly in government contracting. Time and money resources are extremely tight and skillsets are scarce. Mike noted that "ever since the 'Gang of Four' book came out everyone thinks they are an architect and everyone wants to build a framework." I couldn't have agreed more. Ugh, that word: "framework." I've used it many times myself, but if I never see or hear or use it again I wouldn't be unhappy about it.

The purpose of design patterns isn't necessarily to build huge, abstract libraries of software components. The purpose of design patterns isn't necessarily to solve common programming problems for other programmers. A wonderful thing about design patterns is that they are something familiar to rely on when confronted with new problems.

I've used dozens of patterns and I've even participated in writing a framework or two. But the world needs only so many frameworks. At the same time there are no shortages of hard problems to solve for which design patterns are quite, quite useful. I don't have time, money, or expertise for all the frameworks I'd otherwise need to get the job done. I need robust, reliable patterns and people skilled enough to recognize how to implement them to solve customers' problems.

So, attention all developers: Don't just think of design patterns as architectural building blocks. Sure, they are that. But think also of design patterns as tools in a toolbox.

When we're confronted with a domain-specific problem we should build neither point-to-point solutions nor frameworks. Instead, use patterns to solve a domain-specific problem in a reliable way. Point-to-point solutions require niche skillsets, are complicated, and don't scale well. Yet, we don't have time or resources to build frameworks and there wouldn't be much of a market for them if we did. Either that or our frameworks have to be raised up to such a level of abstraction that we end up forcing domain-specific stakeholders into our patterns rather than molding our patterns around domain-specific use cases.

So, what is a good example of how we use patterns to solve domain-specific problems without building a framework? ETLV is a good example:

You don't need a framework to implement this pattern. Simply recognize that the solution to interoperability among domain-specific applications is the production and consumption of well-formed Unicode data. If someone gives you a data source and is looking for a way to visualize it you could look for a visualization API and use point-to-point integration to read the data and constuct objects from the API. But then what do you do when someone hands you another data source? What if the new source lends itself to a new visualization technique? What if the API you chose the first time around doesn't support that technique? Well, perhaps you have job security as a software integrator. On the other hand, if you want to maintain separation of concerns and implement a robust, flexible solution you could follow a pattern based on decoupling data from proprietary or domain-specific formats and transforming data into views using standard, ubiquitous processors. This effectively changes the integration and interoperability problem into a easier-to-solve scalability problem. Point-to-point integration is neither simple nor scalable. However, given any data source expressed as well-formed Unicode we can write a practically boundless number of transformations to produce a practically boundless number of views and applications. It takes a pattern but not a framework.

Models and Simulations as Data

2008-12-09T07:33:00.000-08:00

A colleague, Anthony Watkins, and I recently conducted a feasbility assessment of ETL-V applied to a DoD tactical simulation called JCATS: Joint Conflict and Tactical Simulation. JCATS was created by Lawrence Livermore National Labs in 1997 and has been supported in some fashion or another ever since. JCATS is not unlike other military sims I've encountered over the years; OTB, TDF. In fact, JCATS is generally not unlike all other models and sims with which I have worked in the past 11 years - aerospace apps, power/energy sims, custom apps.

I AM NOT A MODELING AND SIMULATION EXPERT.

I took a class in graduate school, did pretty well. But I studied Computer Science and Applications. That's my slant.

When I reflect on my scant 11 years of experience designing, developing, implementing, and integrating business processes and software - a fair amount of it in the M&S domain - I come to one conclusion.

Interoperability stinks.

A bit of personal history on why that I think that - three words: separation of concerns. As in, we haven't maintained it. I observed the problem in my first job at ODU Research Foundation, Center for Coastal Physical Oceanography: oceanographers were writing code...bad code. Eventually the computer scientists got involved, but all we did was write occasionally elegant code wrapped tightly around a domain application.

Then, in September of 2001, I found Object Serialization, Reflection, and XML all at the same time (XML, casually, in '99). My colleagues and I were using these to create 3D visualization, behavior, and human-computer interaction for the Web. With the introduction of XML-RPC we were communicating with "back end" data services to drive client-side 3D. The 3D applications were entirely dynamic without compilation, the same way all other content was being delivered over HTTP to a standard browser. It was cooler than dirt.

Along came Web Services, circa 2003-4 for us. Finally, everything was starting to click.

That brings me back to models and simulations as data and where I am in 2009.

Simulations aren't applications, they're data sources.

I've written previously about how we use enterprise architecture models as data sources for analytics. It's basically the same thing for sim integration. There is one additional component. Simulations often stream data in real time. The transactional model of the Web and relational databases is not always sufficient. Regardless, the methodology is not dependent on transfer rate or protocol. The methodology is based on separation of concerns. The difference between yesterday and today is that we realized how to take what were doing above the level of a single application.

Most of the applications with which I am experienced are heavy...very heavy. Whether we are talking about footprint, documentation, training, number of participants, facilities, or duration most simulation apps require a lot of it. Technology insertion and training are costly. Learning curves are steep. Exposure and knowledge is limited to very small groups.

It doesn't have to be that way, either.

[Update: 12/10/08 - Well, maybe it does. I appreciate that M&S is hard and hard takes time. The point I really want to make is that we can and should reuse the data generated by M&S to increase their ROI and overall value of their insights as information assets. We do this by decoupling their data so it can be more easily integrated with other things - things people who aren't modelers or enterprise architects care about.]

If I could wave a magic wand, I'd strip every simulation down to it's algorithms and databases. To be sure, there are some sweet front-ends out there, but they aren't maintaining separation of concerns.

JCATS, for example, is a powerful tactical simulation. There are good smarts in there. But JCATS has a limited graphical user interface (2D) and is strictly designed for tactical operational scenarios on maps. While the designers of JCATS may have thought about 3D and some statistical diagnostics, they certainly did not, nor could they have anticipated or accomodated all the many ways we can use the output of JCATS simulations.

The good news is that JCATS saves huge DAT files, i.e., data files, i.e., JCATS data is portable. JCATS produces ordinary delimited text files (comma-separated) and puts similar data on the wire in real time (either TCP/IP or UDP, I think).

From here it's easy:

All of these interactive views are transformations of JCATS data using ETL-V

Government 1.0 -> Government 2.0

2008-12-06T16:53:00.000-08:00

Observing, describing, and defining a 1.0 -> 2.0 transition may be a difficult undertaking. Take Government 2.0, for example. What is our context for "government;" local, state, federal, international? All of these? I think if we are going to attempt any articulation of a 1.0 -> 2.0 transition we should start by following a model. In the case of all things 2.0, that model is obviously the seminal piece by Tim O' Reilly, entitled "What is Web 2.0? Design Patterns and Business Models for the Next Generation of Software."

I suppose we should first start by questioning the appropriateness of trying to draw such inferences from one domain to another. However, this is occuring regardless. The 2.0 moniker is being attached to everything under the sun, perhaps without consideration of anything but brand/buzz recognition. So, like it or not, we are now at a point where we must articulate what we mean by Government 2.0, indeed, by 2.0 in general.

I do think it is appropriate to use Web 2.0 as a model for other domains. Web 2.0 is not just about technology. It is about technology that recognizes and leverages the profound role of human behavior and scale. It certainly seems relevant to Government 2.0

Literally, it may be appropriate to relate the seven top elements of O' Reilly's paper to other domains. Intuitively I can see such relationships in Government 2.0. Harnessing Collective Intelligence strikes me as very Democratic. What could be more E Pluribus Unum? Data is the next Intel Inside is Open Government Data (see also: Kundra). The sub-elements relate as well: "A Platform Beats an Application Every Time" is exactly the point being made by Robinson, et al, in "Government Data and the Invisible Hand."

But even before we get too carried away with that exercise, perhaps we should start where O'Reilly and MediaLive International started; with a brainstorm. What are examples of the 1.0 to 2.0 transition in government?

Here are a few examples that are obvious to me:

NOAA active weather bulletins -> NOAA active weather RSS

Agency reports -> Open Government Data

Budget competition -> transparent investment

representatives' web sites -> representatives' blogs

Public council meeting -> public council WebEx

Requests for Proposals -> Contests for Apps (solutions)

Well, perhaps it's a start anyway.

Enterprise Architectures as Data

2008-12-06T14:35:00.000-08:00

Enterprise architectures are more than static structures, use cases, and process/sequence models. Enterprise architectures are also queryable data sources that, once constructed, can be used to answer a great many questions relevant to decision making based on multiple stakeholder concerns; operational, technical, financial, logistical. In my experience, this assertion usually surprises people. Operators want to know what complex (in their minds complicated) EA models have to do with getting the job done. Appropriators want to know how system producer-consumer dependencies relate to purchasing decisions. To anyone unfamiliar with enterprise architecture, EA can be seen as not only having no value, but as an unwelcome cost burden. Yet, each of these perspectives is relevant to an enterprise architecture. Many architects understand this problem but have been helpless to address it. Enterprise architecture is a rigid, rigorous discipline. The language and views of architects are complex and detailed. The tools architects use are highly specialized. All of this contributes to formidable barrier to information and knowledge sharing.

The good news is that there is a relatively simple technology solution that can cut through the complexity and lead to better decision making informed by a variety of stakeholder perspectives. Now, I'm not saying this is a case of technology riding in on a white horse to save the day, but it's darn close. To be sure, technology's job here is to get out of the way; to provide the least amount of resistance and friction to business processes and people communicating. By focusing on a strategy for how EA data is stored, extracted, and transformed we can make the data more versatile. By making the data more versatile we can make the information that data describe more useable.

The solution strategy, then, is to make enterprise architecture data more versatile using standards.

By implementing this strategy we can use enterprise architectures as data sources to answer a diverse set of typical stakeholder questions. Using this strategy we clearly see that otherwise detailed, complex data can be easily queried, extracted, transformed, and visualized in entirely new ways.

Priorities for Technology and Innovation?

2008-11-22T11:23:00.000-08:00

If I could choose our country's top five strategic priorities for technology and innovation, they would be:

Diversify the energy ecosystem
Transform human transportation
Open more government data
Boost resurgence in manufacturing
Transform farming practices

Diversifying the energy ecosystem means that we diversify production and consumption. Diversifying production means that we increase the number of different means for producing energy in terms of both type and scale; everything from storing the energy you produce in your exercise machine up to expanding nuclear. If they ever figure out the photosynthesis model of using the sun then we're literally set for life. Diversifying energy consumption means that we increase the number of different kinds of energy a consumer can use and decrease the amount of energy the consumer requires in order to be effective. There will always be both high energy and low energy systems that have purpose and value. Efficiency of consumption still matters. Ideally we will produce and require less energy to do more. This is not a pipe dream. This is innovation and efficiency.

Transforming human transportation means that we change the way we think about how people move around geographically. We need to look at the logistics of national and local transportation free from the constraints of mode. When it comes to moving people, we will also place great emphasis on the preferences of the individual. These might be combined into the notion of demand. Given an understanding of demand (past, present, future?) we can make better decisions among biomotive, automotive, locomotive, hydromotive, and aeromotive modes of transportation (most of which are not English words, by the way). I'm not sure I have good answers, but I'm certain transportation is A) complex and B) broken. That usually tells me we need to turn the problem on its head and look at it sideways to recognize what will work.

Opening more goverment data means that local, state, and federal governments and their agencies commit to putting all information already in the public domain online in standard, ubiquitous formats that are easy to publish, syndicate, transform, integrate, and view. Governments should spend less time and money on applications (analysis and visualization tools, websites/portals). Governments should spend more time and overall less money on opening up our data vice building applications. Naturally, government-internal uses of data will require tools and applications. But in all cases better tools and applications will emerge when data is open. This may be the easiest and least costly priority to implement.

Boosting resurgence in manufacturing means that we recreate and reinvigorate our strength and ingenuity as skilled manufacturers. More specifically, we invest in:

High-tech, lightweight manufacturing; tools, devices, sensors, robotics and the like
Transportation manufacturing; planes, trains, and automobiles

Maybe farming equipment becomes an emerging market (see priority 5). Obviously auto manufacturing needs a new direction. Flying is a horrendous and toxic experience on many levels. And trains are poised for a comeback here in the U.S. Clearly transportation of goods is important and deserves attention. But I think transportation of goods is largely solvable while transportation of people seems nearly, but not quite, intractable.

Transforming farming practices means we find the middle ground between niche farming and factory farming. Niche farming is not efficient or practical and is more costly than Americans will endure. The factory farming model is failing in the current environment. There is entirely too much land not being farmed in an effort to control prices and trade, yet prices are high and trade is poor. Some argue that cheap food is gone. I'm not so sure. The reason sited is the high cost of transportation undercutting the (tightly) controlled production practices that created cheap food in the first place. I agree those practices should end. But as long as we are purposely not farming land I don't know how we can make accurate determinations about price realism in food. Here in Virginia and in southerly neighbor North Carolina we have acres and acres of underutilized, fertile farmland. Whether it is land that goes unused because a mid-sized farmer can't compete in a factory-farm environment or old tobacco farms that must transform or die, a new strategy for farming can make use of that land, uplift local economies, keep choices up and prices low.

Then again, what do I know? This is more feeling than fact. I could be terribly, terribly wrong. Clearly I'm short of specifics. What I'm after is strategic priorities. This feels like what matters.

Pentagon Internet Utility Model

2008-11-14T04:40:00.000-08:00

Pentagon gets it right:

Now that you have a space in your cloud to put all your videos, you can come up with much simpler plans for multi-level security in that cloud using technology means that have been thoroughly vetted for decades. Better results with less technology burden. This can solve nearly any of your needs, not only the just-in-time requirements driven by the popularity of personal video production and sharing. Certainly every unclassified activity is covered and that alone makes the world better by several orders of magnitude. But, replicate this model on NIPR, SIPR, and JWICS and every possible thing you could want to do with digital video will be covered by (roughly) the same technology solution. People who have to manage security layers will still have to manage security layers, but that concern is more easily separated from the entanglements of technology now. This is a good thing all around in ways I hope and think you realized before I ever commented here.

This is why you're hot.

More thoughts on Federal Agency Internet Utility Model.

P.S. "OneSource" is a bad term. When it comes to shared content, there is never one source. That's the point. You are the infrastructure, just need to figure out how to make that catchy for PR and marketing. One Destination, maybe. ...needs work in any case.

Pattern Recognition in Graphs

2008-10-24T22:00:00.000-07:00

Recognition of a pattern in a graph is a curious thing. On the one hand, if patterns can be seen among relationships of nodes by their edges, then relationships exist whether or not they can be seen with the eye. In other words, the relationship is inherently mathematical and can be conclusively tested. On the other hand, decision making is not strictly binary, i.e., does the relationship exist? yes/no. Even when decision making is highly quantitative it is most often multi-dimensional. This is where graph visualization comes in. There is significant utility in having the capability to see multiple data dimensions simultaneously by affecting the many attributes of graphical objects. When people are considering multi-dimensional problem spaces then they are engaged in a level of thinking that is poorly supported by low-level primitives (ex., tables of numbers). Decision-making based on tables of tuples is no way to go. Pattern recognition is mind numbing. The best we can do is perform operations on rows and columns; counting, sorting, grouping, filtering. While these are immensly useful and powerful, they overlook the possibility that patterns exist among the atomic, non-aggregate entities themselves. Furthermore, we don't know what they are until we see them and therefore cannot apply operations a priori. On the other hand, given layered, multi-dimensional graphics we can map these many atomic attributes of abstract data to the attributes of graphical objects. Then we can use operations for manipulating graphical attributes to reveal patterns. This is particularly useful in graph visualization, i.e., of networks:

Revealing two distinct communities of interest

BML Hello World

2008-10-02T17:05:00.000-07:00

Loads a 3D Lightwave model:
http://dev.bridgeborn.com/bwschools/BwContent/01_LoadBox.xml

Describes a stacked column chart:
http://dev.bridgeborn.com/bwschools/BwContent/3DChart_SingleRow_Stacked.xml

Describes a force-directed graph (of a caffeine molecule):
http://dev.bridgeborn.com/bwschools/BwContent/ForceDirectedLayout_CaffeineMolecule.xml

Try them out here:
http://dev.bridgeborn.com/bwschools
(IE only right now, sorry)

A Portable Language for Real Time, Interactive, Rich Media

2008-09-29T20:11:00.000-07:00

Following is a draft section from an in-progress whitepaper on ETL-V:

The key to enabling visualization as a core capability is robust description of functionality that is machine portable and parsable through a universal processor that enables the serialization and deserialization of any object, attribute, and behavior in a standard way. Such a description captures the language of visualization in a way that can be understood by any program or person.

Our machine portable language is not intended to express low-level artifacts like vertex points and surface normal vectors, although it can. Most often we follow the model of HTML, which uses URLs to reference images, movies, and other digital multimedia. A 3D Model is not unlike an image:

<img src="images/car.jpg" />

<Model url="objects/car.lwo"/>

Moreover, our language expresses high level artifacts that describe things a person without prior subject-matter expertise might easily recognize. Camera, Model, and Light are but a few examples of rudimentary artifacts in our language. Yet, to establish a robust platform for visualization we must be able to describe artifacts of greater complexity. We must describe the attributes of, for example, ColumnChart, AreaChart, FlowChart, TreeMap, and RadialTree.

Some objects have implied behaviors such as a Sequencer that iterates through a list of items or a Directive that traverses a graph.

Our language must contain more than nouns; it must also contain verbs. Therefore we have command objects like Append, Remove, Interpolate, Morph, Notify, and many others (see also Command schema, below).

Our language must be capable of describing attributes having multiple dimensions, for example a 3-dimensional position. We classify attributes as belonging to one of four basic categories: primitive, complex, compound, and collection. These are defined as follows:

Primitive Attribute – an attribute that is expressed using a single value; ex., integer, floating point number, character; we also consider string to be primitive
Complex Attribute - an attribute that is expressed using multiple primitives; ex., 3D vector of floats
Compound Attribute – an attribute that is itself an Object containing other attributes; Animation has Keyframe attribute, Keyframe is an Object with its own attributes
Collection Attribute - an attribute that represents a data structure, such as a list, vector, or hash map. A Collection attribute can hold any other kind of attribute.

Given this language, we can express any visualization technique, behavior, or interaction style as a human and machine readable document capable of being interpreted at runtime. To demonstrate our concept for this machine portable language, we have created Bridgeworks Markup Language (BML). BML is defined using Extensible Schema Definition language (XSD). Documentation and source schemata are available online in the following locations:

Documentation of BML:

http://dev.bridgeborn.com/Bridgeworks/Schemata/doc/Bridgeworks.html

Schemata: (relative to http://dev.bridgeborn.com/Bridgeworks)
/Schemata/Bridgeworks/Bridgeworks.xsd
/Schemata/Common/Command/Command.xsd
/Schemata/Common/Directive/Directive.xsd
/Schemata/Common/Evaluator/Evaluator.xsd
/Schemata/Common/Node/Node.xsd

[Update 12.02.08: realized the schemata listed above do not contain all of the elements documented...not sure what happened...see BwSchools for full list. Schemata are best viewed in XMLSpy.]

We use Bridgeworks to provide our customers with the largest and most flexible set of visualization capabilities available to them on the market. We deliver these capabilities through a single lightweight runtime that scales between personal devices and large facilities. The runtime engine operates through simple input/output mechanism for parsing text-base, well-formed data, making it easy to integrate without a lot of API programming.

Platform Functional Requirements:

The capability must provide graph layouts (ex., radial tree, cone tree, force-directed graph)
The capability must provide chart layouts (ex., bar charts, line charts)
The capability must provide geospatial layouts (ex., maps, globe, terrains)
The capability must provide high-fidelity 3D modeling and animation
The capability must provide video and audio
The capability must provide hybridized views (ex., Extrude Map: geospatial + chart)
The capability must provide split views (ex., side-by-side comparison)
The capability must provide layered views
The capability must provide a temporal dimension
The capability must provide a mechanism for user-defined attributes
The capability must provide a mechanism for user-defined events, and triggers

Platform Technical Requirements:

The capability must provide multi-dimensional graphics (2D, 3D, 4D+)
The capability must provide an engine that can change views and behaviors at runtime using only a text based, interpreted language that does not require compilation in order to add/modify/delete content in real time
The capability must provide serialization and de-serialization of data and objects into and out of rich media views using:
1. XML
2. JSON
3. CSV
The capability must provide serialization and de-serialization of standard object notations (SONs, ex., JSON) into and out of views
The capability must provide Object/Attribute Reflection
The capability must provide storage of domain data as Objects’ attributes loaded in memory
The capability must provide an extensible API
The capability must provide ECMA script language binding
The capability must provide COM language binding
The capability must provide Java language binding
The capability must be able to render from multiple 2D and 3D file formats, including but not limited to:

ESRI ArcView Shape
LightWave Object
Keyhole Markup Language (KML)
CAD Models (AutoDesk, SolidWorks)
JPEG
Bitmap
Extensible to other formats as needed

The capability must be able to render from data located in local and remote URLs
The capability must embeddable in the following software applications:

Internet Explorer versions 6 & 7
Microsoft Office 2003 Products: Word, Excel, PowerPoint
Mozilla Firefox
Win32 Graphical User Interfaces (GUIs)
.Net/C# GUIs
Java GUIs

The capability must provide an XML schema and a document object model (DOM)

Visualization as a Service

Visualization as a service is an extension of our core capability concept of visualization as a platform. Because we can express views, behaviors, and interaction models without deviating from our standards for reasonable portability we can publish and share views, behaviors, and interaction models just like any other content. This means that web services and other data servers can produce human and machine readable documents that describe visualization, behavior, and interaction. Furthermore, these documents can be parsed to create or modify any visualization, behavior, or interaction technique at runtime without intermediate compilation. Depending on transfer frequency, volume of data, and security constraints, data describing visualizations, behavior, and interaction can also be cached for use in “offline-mode” with so-called “rich Internet applications” (RIAs).

[Update 03.14.2009 - Just watched Tim Berners-Lee's TED Talk on Linked Open Data and recently watched Vint Cerf's Feb 09 lecture to FAA. Want to add here that a document-based approach is the best way to ensure that visualization follows the important lessons taught by these great teachers. Binary objects can be be linked using HTTP and metadata, but binary objects are not open. Binary objects also always require a specialized software reader to be interpreted. Documents, on the other hand, are what the Web was designed to link and text will always be readable by readily available software on the web. Even if one rendering engine disappears it is always possible for another to come along; ex., if rendering an old visualization document as graphics ever becomes important hundreds of years in the future. I realize I am contradicting this principle by encouraging the use of existing binary formats for images and 3D models, but this is a compromise I am willing to make at this point. Check out BML Hello World. KML is important and wonderful, but it is just scratching the surface.]

Federal Agency Internet Utility Model

2008-09-23T16:20:00.001-07:00

Flare gets DDV

2008-09-05T18:46:00.000-07:00

I have learned much from the work Stuart Card and Jeffrey Heer. It was their research on graph layout algorithms that made possible the modularization of graphing capabilities in software. Heer's premier software implementation of this research is Prefuse. Computer Science taught us Graph Theory and the structure, manipulation, and traversal of nodes and edges. Prefuse taught us Radial Tree, Force-directed Graph, and Degree-of-Interest Tree. And Prefuse is excellent not just because it good at graphics and layouts but because it is also document-driven, i.e., it runs on XML. Sure, there's a Java API, and that's probably what most developers focus on. But the fact that it parses XML, that it recognizes a vocabulary and language for visualization, and that this language can be expressed as documents is what I think makes Prefuse special.

After Prefuse came Flare. Flare is a wonderful application of prefuse ported to ActionScript.

Flare really gets document-driven visualization. I was just browsing through the source when I unfolded the data/converters package:

See also (emphases added by me):

	Class	Description
	Converters	Factory class for looking up the appropriate IDataConverter for a given data format.
	DelimitedTextConverter	Converts data between delimited text (e.g., tab delimited) and flare DataSet instances.
	GraphMLConverter	Converts data between GraphML markup and flare DataSet instances.
	JSONConverter	Converts data between JSON (JavaScript Object Notation) strings and flare DataSet instances.

This is certainly familiar.

Tufte on document-based platforms

2008-09-04T09:54:00.000-07:00

This just posted at the end of a new thread on Ask E.T. about Google Chrome:

(Emphasis added by me)

"How's the browser?

Thanks to the many contributors who pointed out Scott's kind link in the example of amazon search.

David Pogue's report in The New York Times suggests this browser leads to an integrated platform with non-proprietary formats (very good) but is still application-based rather than document-based (not very good)."

-- Edward Tufte, September 3, 2008

This is something I think constantly about - document-based platforms can serve end-to-end needs from data extraction, transformation, portability, and visualization. See also, the flow on Document Driven Visualization

Visualization as a Core Capability

2008-08-30T20:35:00.000-07:00

I came back from a recent DARPA workshop ever more convinced that visualization as a core capability is both necessary and attainable. "Visualization as a core capability" means that the full spectrum of visualization techniques is available from any application. It doesn't involve so-called "pluggable" components; no automatic code generation to force big-bang integration between heterogeneous apps, tools, and APIs. Anywhere I come across some columns of data I ought to be able to generate a chart. If I can recognize that columns of data are latitudes and longitudes then the computer should also make that recognition and generate a map. The chart and the map should be independent of any application and at the same time embeddable into any application. I am increasingly of the opinion that we need visualization built into the operating system as a platform service.

Attention DARPA: Here is your National Cyber Range Solution

2008-08-13T20:05:00.000-07:00

Just sayin...

(Image from GWEI.org)

Perma-Knowledge

2008-07-31T19:02:00.001-07:00

I love that people continue to add relevant and useful information to 5 year old threads on Edward Tufte's bulletin board. It happens all the time. Talk about a simple and effective lessons learned resource...

"Information Technology Support" is an Oxymoron

2008-07-28T19:16:00.000-07:00

The purpose of technology is to help people perform tasks. Isn't it ironic that we need so much human support for managing technology? Or maybe oxymoron isn't quite right. "Technology support" is redundant. It's support for support, is it not? I mean, is technology doing its job if it needs lots of ancillary support to keep it going?

Delicious Prestidigital

2008-07-27T17:40:00.001-07:00

Calling All OS Designers: Redesign "Save As..."

2008-06-04T05:28:00.000-07:00

It's time to redux "Save As..." Keyword tagging is everywhere and there is a resurgence in metadata collection, now that benefits are being realized. Indeed applications have included metadata forms for generations but no one ever uses them because they have always been hidden from our awareness and critical task paths. In other words, you have to go out of your way to collect metadata. But this doesn't need to be the case. We learned while working a very complex information organization problem for the FCS Lead Systems Integrator that it is entirely practical to insert minimal steps into the task of bookmarking in order to create a much richer and more easily shared collection. It wasn't a matter of automated versus manual tagging since senior subject matter experts were already using browser-based, hierarchical bookmarking techniques to keep track what they were finding while conducting their research online. (More on that later.) Instead, it was a matter of suggesting that, "Hey, since you are already actively browsing and bookmarking, would you mind if we pop up a window and collect a few key words?" The cost was the number of keystrokes and/or clicks to enter key words plus one to submit a pre-populated tag form.

And so this is what I am suggesting: We should add this kind of metadata collection to the "Save As..." task. Of course it can be opted out, but Save As... is the semantic equivalent of browser-based, hierarchical bookmarking. I won't argue the pros and cons of keyword tagging versus hierarchical cataloging. I know from experience that both have value and that one is much more versatile than the other. I would just ask, how many times do you find yourself saving files with names like:

"Arxiv_electron_force_depth3_foregraph_select_highly-referenced.bmp"

Maybe it's just me and since I can really only speak for myself, I will: I damn sure would enter a few key words during the Save As... process if only I had the means. It needs to be universal, too, or as close as we can get given the hodge-podge of technology on our machines.

I Can't Believe I'm Thinking This...

2008-06-02T19:28:00.000-07:00

I just stumbled across a post from Bruce Sterling on Wired's "Threat Level" blog: "Threat Level is fascinated by the National Cyber Range." I found it because I happened to have read the BAA last week and I was now doing a little homework for a proposal. Let me first record that I applaud Mr. Sterling's calling attention and his quippish commentary. I have to admit that I've had many of the same thoughts over several years of working in this business and even today as I consider implications of potentially working this concept. But given rationalization, a most useful tool and frequent resolver of guilt, I am compelled to counter Mr. Sterling's remarks.

I'll start with the obvious: DARPA invented the Internet; the thing Mr. Sterling is using to exercise his freedom of speech (in the wide open, by the way - something that will get you jailed or killed in other places in the world). Brilliant and now legendary inventors like Vinton Cerf worked for DARPA. Two prominent California universities participated, too. Even then the Internet started with a trepidatious title: "Galactic Network," but here we are.

But enough of the obvious. It is easy to pique our fears by tapping into what is surely a bottomless well of sarcasm and criticism, particularly when one is not bothering to give treatment to any other relevant possibilities. At very least, an entirely contrary position owes some consideration of likelihood. Generally from there we do a good job of protecting ourselves from those downsides, particularly, believe it or not, where our privacy and rights are concerned. A more appropriate criticism would be one that counters the supposed benefits an endeavor such as this program may suggest to offer. Since none are suggested by Mr. Sterling I will have to come up with my own.

I have to start by asking myself, "What existing problems can this solve?" Or, "What questions can this answer?" Or better yet, just plain, "Why?" I know I have to consider these questions in "DARPA-hard" terms. It seems naive to reduce this to a matter of...what? DoD's attempt to CONTROL EVERYONE AND RULE THE WORLD!? ...muhahahaha! (I get sarcasm.) I am absolutely certain that the kind of folks who would work on a program like this are smart enough to know that the concept of people exchanging information in the form of data packets over wavelengths has no constraints on infrastructure. There is always peer-to-peer.

I do suppose it is worth making explicit, by law, the fundamental right to operate your own network on your own infrastructure...

If there was ever an obvious example of a lab experiment having massively unforseen consequences beyond the lab it is the Internet. Ironically, the very spread of the Internet itself must have all of the same characteristics of the spread of an Internet virus or else Internet viruses would lack the very means attributing their basic behavior. Spread of cell phone technology is even more pervasive. And it is all fantastic. But there is some really bad, bad shit that happens on OUR Internet - the one conceived of, invented, and owned by civil people - and it has nothing to do with privacy.

Spam is beyond out of control. It may be intractable. DARPA does intractable. Who would tolerate hundreds or thousands of phone calls daily? Why is Internet spam so disproportionately leveraged as an advertising medium compared to junk mail? Is it because spammers think it's better marketing? Or could it be because criminals are exploiting our Internet for personal gain?

Why would DARPA want the "ability to rapidly generate and integrate replications of new machines?" Could it have anything to do with the existence of multimillion node botnets wreaking havoc?

Then there's this CNN report of researchers demonstrating the feasibility of hacking and destroying critical infrastructure.

So, I just don't know. What exactly is wrong with trying to fix that stuff? Why can't that be DARPA's reason?

How about this one: "The ability to replicate large-scale military and government network enclaves." De facto, the military and government have every right to pursue that ability for themselves if they think they need it. Er, ourselves...if we think we need it. Whether we like it or not the military and the government is us and us and us. But, hey, wouldn't it be great if DARPA invented a revolutionary new way to replicate [FOO] large-scale network? Like the ones on which our banks run.

And this one: "The ability to connect to distributed, custom facilities and/or capabilities as necessary to incorporate specialized capabilities, effects, or infrastructures." Hello, system integrators. DARPA should call Apple for this one.

And: "Realistically replicate human behavior and frailties." Well, that's just DARPA being DARPA-weird.

Wonderfully Simple Praise of the Hyperlink?

2008-05-08T19:46:00.001-07:00

(Prologue: This title comes from posts written by Ben Fry and Jeremy Keith. I hope they will not mind. I find myself increasingly attracted to what they have to say on technology matters I care about.)

Here is near perfect example of a complexity problem, that I have yet to define, when comparing the integration of two APIs: Facebook and del.icio.us. One API seems to indicate what happens when you let developers define use cases and one shows what happens when you actually stop to think about how information is used by consumers. (I can say that because I write code.)

So in this case, I want to get a list of my friends. It's a dead simple request. I tell the platform I want my friends. I get back a well-formed (hopefully) document containing my data about my friends, right?

Exhibit A, The Facebook API:

facebook.friends.get

The name suggests a clean, intuitive interface...until we get into how we use the interface.

facebook.friends.get has nine parameters. Two of the parameters are separate authentication values.

The parameter call_id is documented as follows:

"The request's sequence number. Each successive call for any session must use a sequence number greater than the last. We suggest using the current time in milliseconds, such as PHP's microtime(true) function."

Which is to say that the caller is responsible for scheduling calls with the service?! Why do we have to agree on a "sequence number" in order to do business?

How about the "variable" that isn't:

string v This must be set to 1.0 to use this version of the API.

Which is to say, "Please pass me this parameter even though I already know the value."

It's a "placeholder," right? Because in "the future" we might want to leave room for specifying which version we want...per interface.

It's goes on and on. Then you tell facebook.friends.get what format you want, then whether or not you want the results wrapped in a function call so you can get around cross-domain data security, yada, yada... And don't forget to hash the URL first! I'm only talking about one endpoint here.

Ugh.

Am I wrong to be disappointed? Am I the only one who is confused? Who thinks this is unnecessary? This is the standard API-centric view of interoperability but I'm not buying it.

Now, I have little doubt there is a good reason for every parameter and for this kind of programming. I've heard every reason. I argued many of them myself...until I learned better. To gnash teeth over the legitimacy of each thing is to argue details regardless of their relevancy to the matter at hand. The matter at hand is that I am requesting my friends from the platform and I am expecting back a well-formed document as a result. That's all. From here the platform should do everything it possibly can limit the amount of additional information it requires of me. I'll grant that platform expects me to authenticate my request.

The problem with facebook.friends.get is that it is trying to serve multiple use cases through a single request, as if that is more efficient or more simple. Most of the use cases serve developers (or so they may think). Most use cases are more easily supported than the basic task I am trying to perform. Remember? It's "get my friends. " I find myself restating this important detail like a mantra in order for it to have a greater voice. Why is the service burdening the consumer with all of this minutia for a simple data call that should return a document?

Exhibit B: The del.icio.us interface for the same data:

http://del.icio.us/feeds/json/network/prestidigital

It's one simple URL. It returns well-formed data as JSON. Oh, the simplicity! Oh, the possibilities!

Detractors would no doubt point to the fact that this URL is neither authenticated nor encrypted. I would counter: that was a choice made to serve customers in this case as delicious certainly supports authenticated, encrypted interfaces that are just as easily used as /feeds/json/network. Beyond that, everything else is a decoration.

And, OK, so developers do need more options and features than can be provided through a simple URL interface. But those should be added as one or more layers on top of a layer that just provides well-formed data. I'll accept that if someone can show me: where's the layer that provides the data? It's obfuscated by the business layer. This is no doubt an artifact of the traditional three-tiered architecture (a model of which I am a proponent by the way).

On the Internet, the way in which data is located is via the URI and the way it is passed is HTTP. It's rudimentary: every container of data has a uniquely addressable identifier. Web service endpoints are URLs. We need to get developers and Internet platform providers back to this fundamental principle.

Exhibit C: Yahoo! Internet Location Platform:

http://where.yahooapis.com/v1/place/2507854

Click the URL, get back an XML document. Now THAT'S data portability; that's interoperability.

Update 11.17.09: BUMMER, this link is broken. It looks like an auth issue.

Best Practices of Well-Formed Data?

2008-04-25T20:18:00.001-07:00

Update 2008/05/06: It occurs to me now that this works for me due to my dependence on Western (American English) style characters: http://benfry.com/writing/archives/82... My concern is readability by humans and for me that means American English. Unicode is a super set, so I have modified my post accordingly.

I have come to realize that there are exactly three (3) formats for well-formed data which suit all of my interoperability needs:

XML
~~ASCII~~ Unicode standard object notation

(i.e., script object notation, i.e., SON, i.e., Object.property as text, ex. JSON)

~~ASCII~~ Unicode standard delimited notation

(ex., comma-separated values)

Each of these has upsides and downsides.

Use XML when you want to be the most versatile with your data. Besides being ubiquitous, XML serializes and deserializes quickly and easily, represents OO clearly, and can be easily, automatically validated. XML presents itself well for both people and software. Especially use XML when you want to do advanced searches on data or transform the data from one domain-specific language (DSL) to another. DOM and XPath queries do lose their utility as data sets get large, but these situations can be avoided and parsing remains fast even on files many megabytes in size. Batch processing gigs of data is not a job for XML. (Use a relational database for that.)

Use SON when all you want to do is serialize/deserialize data to/from business objects or standard data structures (ex., arrays, maps). JavaScript Object Notation, or JSON, is super convenient for Javascript apps and JSON "liblets" (very small libraries, usually only 1 file) are easily found and maintained with your own code. SON also represents OO clearly, and given a pretty-print function, presents itself well to both people and software. But, there is currently no way to validate a SON. Not everything can handle a SON either. ~~JSON doesn't work with languages that aren't JavaScript~~*, although there's nothing about JSON that is proprietary to Javascript so it's just a matter of time. JSON is just formatted text. ~~ASCII~~ Unicode-based SONs may one day be the XML killer but that time is a long way off.

Use CSVs when you want to be super fast on the throughput and can make assumptions that the data is simple to interpret. CSVs work well when moving very large amounts of data over a wire and/or into and out of relational databases. CSV data does not represent OO in any way, it does not present itself well to people, and there is no standard way to automatically validate it. CSV can be super efficient, but it is error-prone, can be difficult to debug, and is not nearly as versatile as the other two formats.

A good data service should support all three of these whenever and wherever it is feasible. And here is the obvious point that I do not want to escape from me - these 3 formats are all any data service ever need provide. Any service that supports each of these will be "universally" (practically) interoperable. If all data services supported exactly these 3 then all the Web would be an oyster.

*Update: over at JSON.org you can find ports for every language under the sun. Actually, what I meant is that support for JSON isn't native in the main platforms. Hopefully it will be built into all of these languages in the future.

Is Google Search Appliance Appropriate for Government and Corporate Enterprises?

2008-04-06T20:40:00.000-07:00

Question to Ponder (QTP)

Google's smarts are based on Page Rank, which is tied to popularity. Popularity on Google is heavily influenced by financial and other driving incentives. Google's strengths are based on Gi-normous machine clusters and on huge critical mass of the widest diversity of users, who are mixing it up and providing feed back into the system explicitly and implicitly by way of their behavior.

So is a Google search appliance a smart recommendation for government or corporate enterprise? I don't know exactly but last good info I had was that they cost about $150K per appliance rack that could search 2 or 3M items. [Update 01.14.2009 - new box searches 10M docs] One NASA center has two racks and found they chewed up 4-6M items in no time. I don't know what percentage that was, but I do know that they were want for more racks. Last I talked to them they were getting good at setting up filters to weed stuff out. (Weeding the garden is probably not a bad thing in any event.)

Now, I suppose you could - antitrust laws and fear not withstanding - scale Google's model to apply to the whole of the US federal government's information infrastructure and we'd approach some aspect of critical mass. Popularity behavior would surely be different than the commercial advertising model (I think). But that's not necessarily bad, if true.

It actually might not cost a lot of money in terms of federal budget scales. The way I figure, 1 billion records would cost $50M at full retail cost, i.e., no negotiated discount:

1,000,000,000 items/3,000,000 items per rack = 333 racks : 333*150,000 = 50,000,000

Tack on 20% to implement? +10M gets us to $60M
20% per year for maintenance? +10M per year * 10 years and we're at $160M

Double it all for 2B items. $320M. Even with no write downs for discount that's a rounding error in the federal budget.

To avoid monoply and other sticky issues with a deal like this, the government should buy Google's intellectual property and arrange a reseller license agreement so the government can contract out to the fairest qualified bidders (plural) the secure right to implement the IP on hardware on the government's behalf. Google could even be forbidden from competing in that way for some number of years. Government then leases time on the GIG to cover the bandwidth resources consumed by the huge clusters serving this purpose. (Ooh - is that in the cost?) Government already has the ability to secure this at multiple levels of security (Is that in the cost?). It is essentially, if perhaps not trivially, a matter of organizing search clusters to operate within NIPR or within SIPR or within JWICS firewalls.

[Update on 11 April based on conversation with Katharine: It might not be one big Google search farm pointed at these networks, at least not without a radical change in policy that may be intractable due to politics. There are many, many organizational firewalls. But at least get the bulk purchase order in, get a huge discount...for taxpayers, and place clusters where they need to be to get the biggest bang for those bucks. The essence is to take a strategic approach to search for the whole of government information. Search is so very fundamental that it deserves this kind of treatment.]

Ok...that demon thought is expelled now. I'm writing too many proposals. Approaching temporary burnout threshold. My brainstorming synapses may be totally misfiring at this point.

[Update 3 December 08: 2B searchable items is nary a drop in the bucket, I reckon. Are we talking about 2 trillion? 200 trillion? Large numbers that sound obscene when spoken?]

On Accessors and Mutators

2008-04-02T00:17:00.000-07:00

I was just exploring the Google Visualization API reference and found myself a little hung up something. It's not just Google's API, but they are certainly a huge propagator of code. It's not the first time I've been hung up on this, either. The problem is that most APIs that use the Accessor/Mutator pattern follow it in the same, and in my opinion wrong, way. What is the point of declaring and implementing a unique function - no - two functions for every property contained by an Object? In fact why do we need any more than exactly two functions total to get or set any property or, in fact, any Object?

getColor()
getName()
getNumberOfColumns()
getNumberOfRows()

Why not just get(namedAttribute) and set(namedAttribute, value)?


get("color")
get("name")
get("numberOfColumns")
get("numberOfRows")
set("numberOfRows", 6);

Realize that 6 is probably not the literal integer 6, but rather an Object itself or perhaps a Collection (of size 1) containing literal, primitively typed values. But that is hardly of consequence in the scheme of things, it seems to me. I dread even having to make the case for efficiency. Programmers still get hung up on efficiency too much in terms of cycles and RAM. While I am sensitive to these concerns (for reasons of achieving enough throughput to support fluid user interaction) such concerns pale in comparison to what is lost overall to complexity of integration. How many API's do we want to learn, really? Domain-specific languages are much easier to learn and use, and more so when there aren't too many competing DSLs floating around a domain.

Perhaps I have it wrong and it is all a trivial point I am making. But it seems an awful waste of time doing it the other way. What kind of code bloat is being propagated? What happens when you add a new attribute? In fact, the ability to declare an arbitrary property and assign it to any Object is a powerful mechanism in software. It will be used and abused one way or the other so we should anticipate and handle it. But specifically, this kind of abstract polymorphism is ideal for serialization and deserialization of database records without having to tightly bind classes to database constructs through object relational mapping. Whether using XML, standard object notation, or standard delimited formats, there are multiple ways to deserialize any Object into memory for use by business logic without having to declare and implement a class for the Object.

In fact, one interface for get and one for set is much more flexible on the whole. Integration and interoperability with other programs is much easier with a single get/set. The ability for interoperating software programs to bind solely to a universal input/output protocol, and for interoperating software to remain blissfully unconcerned with the inner workings of peers is beyond powerful. Then we can focus on the functional roles of software. What does it do? I don't care how (beyond the I/O interface)(and I do care how well). When you get to that point, software can program other software (within a security model, of course). This kind of integration is also easier to maintain. Consider an implementation that uses XML as the carrier. The main Do-Until-Error loop is as simple as: While(Element is recognized) Process Element; Else Throw/Ignore (Unrecognized Element Error). A secondary error condition will exist in the event that expected inputs are no longer generated as outputs from peers*. In both cases, the situations are reduced to a matter of choosing how to process output from peers as it is provided and not also a matter of refactoring the interface between peers. It's all the UNIX command line applied to the Web. And I am certainly not the first or only person to have noticed.

* A third condition occurs when input is malformed

CAVE ETD

2008-02-25T17:01:00.000-08:00

This movie (above) was a product of a class assignment from graduate school, co-developed by Hussein Suleman and Fernando Das Neves. Talented guys. Actually, they did most of the development. I was the CAVE SME but still a fairly green agile developer. The class was Dr. Ed Fox's Information Storage and Retrieval (archived). The assignment was a basic proof-of-concept for navigating the Electronic Thesis and Disse rtation (ETD) digital library at Virginia Tech using a 3d virtual reality metaphor. This is probably first time I knew for sure that realistic physical metaphors for browsing large electronic collections is a complete waste of time. It's even worse when the digital collection does not map to any actual physical space (i.e., this isn't a model of the campus library) and that's usually the case. On the other hand, tagging virtual environments with contextual information can be quite useful. Keep in mind that this is screen capture from monitor, but we are in a 10 cu. ft room, fully immersed in a 3D environment. So if you then keep everything the same but change the use case...there might very well be some value in linking the contents of a digital library to a virtual environment as a way of enriching the environment. Instead of wandering through a library having a completely arbitrary "floorplan" conceived by the application developers you are looking at a piece of complex equipment inside a virtual power plant and you can click anywhere and link into the: parts catalog, component specification, training lesson...any digital content. That would be more useful.

Visualization Doesn't Matter

2008-02-15T19:48:00.000-08:00

Sense-making matters. I've always been cognizant of the fact that people without sight have exactly zero use for visualization. (Probably not 100% true, I bet.) The point is that there is nothing presumptive about visualization that makes sense-making possible. It is the human mind that makes sense-making have even meaning enough to give way to possibility. Communication matters, too. Certainly visualization makes communication possible by providing an entire medium/channel/wavelength such that when we lack it our communication may likely be impeded.

Obviously I think there is value in visualization. So why do I say it doesn't matter? Because visualization is never more than a means to an end. That end could be to have fun or it could be to make an important decision or it could be to send a message. But in any case, there can be 1001 means to achieving those ends and visualization is never more than one of them.

Why is this point worth understanding and remembering?

Selling. Working for customers. Providing a service to people. I am a computer scientist and a technologist and a visualization "guy" but my work and my business are all about people. Sometimes people forget what the word "technology" means.

And so when I'm talking to a customer about visualization and I find that we are talking too much about visualization then something is not right. We should be talking about the customer's wants, needs, pains in intellectual terms that are not constrained by any one means of achieving an end. It's up to me to decide how, where, why, and when visualization matters. That's what I'm getting paid to do.

Ok, so enough with the hand-waving. What is a practical example?

I have a customer who wants and needs to make an investment in visualization and part of his information technology capability. I know going in that the reason I am doing business with this customer is because he values visualization and knows he needs help integrating it into his IT portfolio. So I'm sitting down in front of him and giving him his first brief on what I am doing for him. Most of the hour long presentation (demonstrations, actually) is/are in terms of how visualization is used to present and interact with his domain data. He finds that all well and good in about 5 minutes, but what he really wants to know is "which of my problems does it solve?" And that's where it gets tricky. Because the answer is that visualizing domain information doesn't solve any problems de facto.

So do you want to talk about my visualization services or do you want to talk about your problems? What matters? I'm happy and comfortable discussing either. But in my business, people solve problems and it just so happens that my people are skilled at using visualization as a means toward that end.

Social Network Portability or Portable Social Networks?

2008-01-31T08:10:00.000-08:00

I find it wickedly ironic that all the worthwhile activity around social network portability and data portability is trumped by ordinary human behavior:

Note that what Julian asserts there is valid, in my opinion: a whole lot of thought and knowledge, both good and bad, was poured into SNP in the "early days." If the people have moved from one group to another then perhaps it is safe to assume the knowledge went with them. But if this group died and the other emerged...then what?

And if that weren't confusion enough:

So here we are trying to keep up with the conversation - which can be extremely difficult in active groups - and the conversation keeps moving...figuratively and literally.

"Wait. It's not just social network portability. It's ALL data portability. Let's all move up to data portability."

"But 'all' data portability is too big. There are too many facets. Let's all move down into the various facets (presumably according to our interests and perspectives)."

Down that path madness lies if one isn't careful. Managing the order of things when it comes to the Internet can drive one crazy enough as is.

I do admire them for trying. Progress is being made. Data portability is intensely important for connected computing and process interoperability.

Latest g.licious

2008-01-08T20:50:00.000-08:00

More noodling around...

From Bridgeworks

On API Integration

2007-12-20T08:38:00.001-08:00

This quote from Danny Ayers in the social-network-portability group is spot on and goes a long way to help explain why our approach to document-driven visualization is the best way to integrate:

Architecturally, there are at least two big advantages in taking this
approach [of import/export to common portable formats].
When we have an environment which contains N different formats/
APIs which can express the social information, either we need
N^2 different converters to enable interop *or* we can map to a common
model, and just need N different parsers/serialisers - still
non-trivial, but a lot easier. The second advantage, which is the
reason an RDF-backed is appropriate, is having the flexibility to
compatibly include whatever (other) information you like in the common
model, beyond the limitations of the typical individual formats/APIs.
Check Tim Berners-Lee's FOAF [6].

all.my.base

2007-12-19T04:29:00.000-08:00

I wanted to see if I could remember/find all of the web spaces where I have a profile and/or social network:

Linked In (as kevincurry)

del.icio.us (as prestidigital)

facebook (as kmcurry)

Last.fm (as prestidigital)

My Space (as prestidigital)

Yahoo! (as prestidigital)

OpenId (as kevincurry)

Simpy (as prestidigital)

Slashdot (as prestidigital)

TED (as KevinCurry)

Orkut (as Kevin Curry)

MSN (as hokiestoked)

AOL (as hokiestoked)

Update 01/06/08 - I'm updating this list in the butter room

Net-centricity v. Social Networking

2007-12-04T07:50:00.000-08:00

Danger Room Editor, Noah Schachtman, has a rather interesting article on this topic in the Dec. 2007 print issue of Wired, 15-12. The main theme is the juxtaposition between net-centricity and social networking (although not exactly the social networking we know through Web 2.0 popular culture). The message about net-centricity is that it is highly effective as a destructive, lethal capability while social networking is better at stability operations and counter-insurgency. Some of the most advanced social networking groups in Afghanistan are just starting to step it up with computer hardware and software, I reckon to enable things like social bookmarking, collaborative placemarking, network analysis, pattern matching, and the like. But the focus in Iraq and in general is on personal interaction, almost eschewing technology. The irony of ironies is that our military has an excellent solution to a problem it does not currently have (fighting big wars with nations and dictators), while the terrorists and insurgents have swiftly mastered the technology domain of modern social networking - email, cell phones, text messages, chat, forums - and the military is reportedly now most successful resorting back to low-tech, human/psychological solutions. Not coincidentally or surprisingly, there is some heated debate about the role of social science in military operations. My interest is in the heart of the matter...the balance between communication technology and human interaction, particularly when technology gets in the way or solves the wrong problem. The insight in the Wired article is that, technically speaking, a better application of IT to the military's current and future challenges related to stability ops and counter-insurgency, as previously inferred, is closer to Web 2.0 social networking and collaborative read/write web technologies, web services, pub/sub and the like. I would go so far as to project the need onto the whole of government and civic life - federal, state, and local.

Net-centricity

2007-11-19T05:49:00.000-08:00

I received a forwarded email from a colleague entitled "DoD seeks net-centricity help." I found a copy here. It seems from this article, at least, that they do in fact have a problem articulating their message. I also suspect they aren't talking to the right...vendors. Mr. Montemarano is sparse on specifics and fails to bound the problem. I know it's just a syndicated news article, but it's the message I received. (Ironic, isn't it?) What do you want me to consider? He says he wants vendors to offer pub/sub. Ok, great. Done. But, what about security? Presumably that is important to DoD. What about the Army's mobile, ad-hoc requirements for FCS? Net-centricity is a broad concept. Some argue that it also frames the problem in terms of technology and not people. There is certainly something to that. As a career student of computer-supportive cooperative work (CSCW) and human-computer interaction (HCI), I usually defend that position. (Consider the case here, where news with the wrong message has been successfully syndicated to a wide audience. Tech did it's job.) But sometimes it is just about the technology and I think that is the case with DoD net-centricity in the way I infer here. I would agree that vendors are not implementing what Mr. Montemarano seems to need. DARPA invented the Internet. It was an amazing technical feat. Communication-wise it was about as complicated as "Hi." But DARPA must have become a one-hit wonder in the telecom domain. What Mr. Montemarano wants is Web 2.0. And he can have it. Want is not happening is that no one is seriously considering how to get Web 2.0 done in what are rigidly constrained environments inside DoD. And O' Reilly's original essay on Web 2.0 isn't without flaws. "Wisdom of Crowds" can also lead to "Group Think" and gross disinformation. Mr. Montemarano, my name is Kevin Curry. I work for a company called Bridgeborn. We have offices in Virginia Beach, VA, Arlington, VA, and Savannah, GA. Give me a call.

On Addressability of Data on the Web

2007-11-15T14:32:00.000-08:00

I don't understand data sharing solutions that require centralization of data stores. Centralization is an organizational artifact, not a technical one. "The way to solve this is...everyone stop putting you data there and start putting it here." By the very nature of the web, this requirement will never be fully satisfied. There will always be something not in the "master database" that is relevant. And how do you consume data from multiple sources, i.e., multiple perspectives (ex., operational, financial, organizational)? Centralization is seldom even possible, by design, in classified environments and highly improbable when sharing data potentially leads to losing your budget. If data exists on the web and needs to be shared then the way to share is clearly pub/sub; to syndicate query results over HTTP. Probably, but not necessarily, XML will be used to carry data. Technically this should be a no-brainer. Wrap database stored procedures with methods on a Web Service and point to them with URLs. Everything we share on the web we share over HTTP using URLs for addressability. Addresses are fundamental to how data is managed in computers down to the hardware. Why should data on the web be any different? They aren't:

[Note: These links are illustrative. They don't work]

GetConsumerOpNodesByCountOfIER
Returns an Xml Node containing the list of Consumer Operational Nodes and their IER counts, ordered by count
GetConsumerSysEntitiesByCountOfIER
Returns an Xml Node containing the list of unique System Entities ordered by number of IERS for which the SysEntity is used as a Consumer
GetConsumerSysNodesByCountOfIER
Returns an Xml Node containing the list of Consumer System Nodes and their IER counts, ordered by count
GetMissionThreads
Returns an Xml Node containing the list of unique Mission Threads
GetMissionThreadsByCount
Returns an Xml Node containing the list of unique Mission Threads by count of threads
GetMissionThreadsForOpNode
Returns an Xml Node containing the list of unique Mission Threads in which the given OpNode participates.
GetOpActivities
Returns an Xml Node containing the list of unique Operational Activities

The results of these queries are usable by both people and software.

Sidebar: Just like hardware, data addresses aren't particularly people-friendly. Look to the ideas behind semantic web to help with that.

Form-driven Editors and Language as Cognitive Artifacts

2007-11-12T21:04:00.000-08:00

I was recently noodling in an Access database, creating queries. By default the "New Query" dialog steers users to a form-driven editor. It takes a couple of clicks to get to a text editor for writing SQL.

When writing SQL, at least, I'd much prefer the text editor by default. The reason is that the form editor slows me down at a very fundamental level. It's not as if I'm a coder with a chip on my shoulder about graphical code generators. Far from it. I can think of at least two that are quite handy at what they do [1, 2]. But with a language like SQL it's always going to be easier to learn how to speak the language than learn how to use a proprietary GUI. Even if Access was the only database technology I ever used (which is not the case), I know intrinsically that SQL is the right way to query a database and a form editor is the wrong way to query a database. The reason is that the act of querying a database is, at its core, an act of language expression. Filling in a form and dragging boxes around are acts of interaction that get in the way of expression...of me telling the computer what I want it to do.

This must be the inherent lesson from Donald Norman's theories on cognitive artifacts. It's interesting that I found this article by him just now. It says:

"And of all the artifacts that have aided cognition, the most important is the development of writing, or more properly, of notational systems: number systems, writing, calendars, notational systems for mathematics, engineering, music and dance."

Visualizing My Email

2007-10-07T17:48:00.001-07:00

Just getting started,
but the initial results look interesting and promising.

Not An All Or None Situation

2007-10-04T04:48:00.001-07:00

Document-driven visualization represents a disruptively powerful innovation for visualization interoperability. It means that we can treat visualization in the same way that we treat the clear majority of data on the web.

But I'm having a mini-reawakening to the bigger picture in which we see the value in APIs and ultra-lean (and more secure) data transfer. When we tie into simulators we still use XML to configure the system, but in a way that tells it to use the API to connect to values directly as they come off a TCP/IP socket connection. The same is true for online collaboration. The object in these cases is to move data efficiently and reliably. There's no need to pull in all the DOM overhead and seldom any need to perform transformation of the data. (But even complex rules for converting Attribute types can be encoded into Objects that are then configurable through Reflection, Serialization, and ultimately XML. We call them "AttributeConverters.") The other network protocols (TCP/IP, UDP...not really FTP) contribute by giving us a lot more flexibility in terms of quality of service. Again, OO fundamentals are key. Object Reflection means we can get and set any attribute value in a standard way...in the API. Any integration more complicated than that usually indicates that our core visualization capability is missing something. Ultimately we want to move closer to the GPU...I think...dual core processors also represent a new (finally stable) option for better performance laptops and PCs.

I digress. Document-driven viz is super neat, but it isn't everything. I suppose I'm just echoing a common theme in life. Not a Panacea. Not a Silverbullet. Not A Cure for World Hunger. Just the best thing I've seen in my brief career.

Thoughts on secure data publishing and transformation governance and best practices

2007-10-03T06:39:00.000-07:00

I'm not a security guy, so maybe I'm out a limb here. Douglas Crockford recently called the DOM and huge security problem. I'm looking at a huge traceability and lack-of-symmetry problem. The problem I usually run into in big enterprises is not with permission to trace data, but the complete in ability to do so. When people can't trace data back to sources and when transformation processes are asymmetrical, it becomes extremely, exhorbitantly difficult to make sense from information; V&V goes out the door.

Claim:

Data exports should pass metadata describing, at minimum, their immediate source, and any transformation applied by that source when producing the data. This metadata forms a singly-linked, directed list from data targets to their sources (ex., from an Excel spreadsheet back to a System Architect model). The link list can be followed only by persons and systems having clearance to do so.

Motivation:

Better traceability and symmetry in the data transformation and publishing process

Specification for metadata:

Required field: URL for source

Required field: Security Classification of source

Required field: Security Classification of metadata and traceability permission

Required field: Timestamp

Required field: Author, publisher name

Optional field: Name (aliases?) of source

Optional field: Name (aliases?) of transformation

Optional field: URL for transformation, when used (bad practice to not include when present)

Optional field: Description of source

Optional field: Description of transformation

Technical Requirements:

It is the producer's responsibility to describe itself by placing metadata into the exported target's DOM. This is not a "must" requirement. Data producers have the right to be anonymous.

Use Cases:

Cleared receivers of data can trace data back to the data's source by following a hyperlink to the source, or to metadata when the source is incapable of acting as an Internet asset. The link chain can be followed as long as the tracer has clearance to go to the previous node, i.e., to visit a URL. The burden for granting or denying access to URLs is already maintained by existing systems (ex., Web server permissions, CAC).

Traceability is stopped by denying permission to get to the URL for the source .

Metadata is scrubbed by pruning its node(s) from target's DOM.