Showing posts with label integration. Show all posts
Showing posts with label integration. Show all posts

Tuesday, December 9, 2008

Models and Simulations as Data

A colleague, Anthony Watkins, and I recently conducted a feasbility assessment of ETL-V applied to a DoD tactical simulation called JCATS:  Joint Conflict and Tactical Simulation.  JCATS was created by Lawrence Livermore National Labs in 1997 and has been supported in some fashion or another ever since.  JCATS is not unlike other military sims I've encountered over the years; OTB, TDF.  In fact, JCATS is generally not unlike all other models and sims with which I have worked in the past 11 years - aerospace apps, power/energy sims, custom apps.

I AM NOT A MODELING AND SIMULATION EXPERT.  

I took a class in graduate school, did pretty well.  But I studied Computer Science and Applications.  That's my slant.

When I reflect on my scant 11 years of experience designing, developing, implementing, and integrating business processes and software - a fair amount of it in the M&S domain - I come to one conclusion.

Interoperability stinks.

A bit of personal history on why that I think that - three words:  separation of concerns.  As in, we haven't maintained it.  I observed the problem in my first job at ODU Research Foundation, Center for Coastal Physical Oceanography:  oceanographers were writing code...bad code.  Eventually the computer scientists got involved, but all we did was write occasionally elegant code wrapped tightly around a domain application.

Then, in September of 2001, I found Object Serialization, Reflection, and XML all at the same time (XML, casually, in '99).  My colleagues and I were using these to create 3D visualization, behavior, and human-computer interaction for the Web.  With the introduction of XML-RPC we were communicating with "back end" data services to drive client-side 3D.  The 3D applications were entirely dynamic without compilation,  the same way all other content was being delivered over HTTP to a standard browser.  It was cooler than dirt.  

Along came Web Services, circa 2003-4 for us. Finally, everything was starting to click.

That brings me back to models and simulations as data and where I am in 2009.  

Simulations aren't applications, they're data sources.

I've written previously about how we use enterprise architecture models as data sources for analytics.  It's basically the same thing for sim integration.  There is one additional component.  Simulations often stream data in real time.  The transactional model of the Web and relational databases is not always sufficient.  Regardless, the methodology is not dependent on transfer rate or protocol.  The methodology is based on separation of concerns.  The difference between yesterday and today is that we realized how to take what were doing above the level of a single application.

Most of the applications with which I am experienced are heavy...very heavy.  Whether we are talking about footprint, documentation, training, number of participants, facilities, or duration most simulation apps require a lot of it.  Technology insertion and training are costly.  Learning curves are steep.  Exposure and knowledge is limited to very small groups.

It doesn't have to be that way, either.

[Update:  12/10/08 - Well, maybe it does.  I appreciate that M&S is hard and hard takes time.  The point I really want to make is that we can and should reuse the data generated by M&S to increase their ROI and overall value of their insights as information assets.  We do this by decoupling their data so it can be more easily integrated with other things - things people who aren't modelers or enterprise architects care about.]

If I could wave a magic wand, I'd strip every simulation down to it's algorithms and databases.  To be sure, there are some sweet front-ends out there, but they aren't maintaining separation of concerns.

JCATS, for example, is a powerful tactical simulation.  There are good smarts in there.  But JCATS has a limited graphical user interface (2D) and is strictly designed for tactical operational scenarios on maps.  While the designers of JCATS may have thought about 3D and some statistical diagnostics, they certainly did not, nor could they have anticipated or accomodated all the many ways we can use the output of JCATS simulations.

The good news is that JCATS saves huge DAT files, i.e., data files, i.e., JCATS data is portable.  JCATS produces ordinary delimited text files (comma-separated) and puts similar data on the wire in real time (either TCP/IP or UDP, I think).

From here it's easy:
















All of these interactive views are transformations of JCATS data using ETL-V



Sunday, April 6, 2008

Is Google Search Appliance Appropriate for Government and Corporate Enterprises?

Question to Ponder (QTP)

Google's smarts are based on Page Rank, which is tied to popularity. Popularity on Google is heavily influenced by financial and other driving incentives. Google's strengths are based on Gi-normous machine clusters and on huge critical mass of the widest diversity of users, who are mixing it up and providing feed back into the system explicitly and implicitly by way of their behavior.

So is a Google search appliance a smart recommendation for government or corporate enterprise? I don't know exactly but last good info I had was that they cost about $150K per appliance rack that could search 2 or 3M items.  [Update 01.14.2009 - new box searches 10M docs]  One NASA center has two racks and found they chewed up 4-6M items in no time. I don't know what percentage that was, but I do know that they were want for more racks. Last I talked to them they were getting good at setting up filters to weed stuff out. (Weeding the garden is probably not a bad thing in any event.)

Now, I suppose you could - antitrust laws and fear not withstanding - scale Google's model to apply to the whole of the US federal government's information infrastructure and we'd approach some aspect of critical mass. Popularity behavior would surely be different than the commercial advertising model (I think). But that's not necessarily bad, if true.

It actually might not cost a lot of money in terms of federal budget scales. The way I figure, 1 billion records would cost $50M at full retail cost, i.e., no negotiated discount:

1,000,000,000 items/3,000,000 items per rack = 333 racks : 333*150,000 = 50,000,000

Tack on 20% to implement? +10M gets us to $60M
20% per year for maintenance? +10M per year * 10 years and we're at $160M

Double it all for 2B items. $320M. Even with no write downs for discount that's a rounding error in the federal budget.

To avoid monoply and other sticky issues with a deal like this, the government should buy Google's intellectual property and arrange a reseller license agreement so the government can contract out to the fairest qualified bidders (plural) the secure right to implement the IP on hardware on the government's behalf. Google could even be forbidden from competing in that way for some number of years. Government then leases time on the GIG to cover the bandwidth resources consumed by the huge clusters serving this purpose. (Ooh - is that in the cost?) Government already has the ability to secure this at multiple levels of security (Is that in the cost?). It is essentially, if perhaps not trivially, a matter of organizing search clusters to operate within NIPR or within SIPR or within JWICS firewalls.

[Update on 11 April based on conversation with Katharine: It might not be one big Google search farm pointed at these networks, at least not without a radical change in policy that may be intractable due to politics. There are many, many organizational firewalls. But at least get the bulk purchase order in, get a huge discount...for taxpayers, and place clusters where they need to be to get the biggest bang for those bucks. The essence is to take a strategic approach to search for the whole of government information. Search is so very fundamental that it deserves this kind of treatment.]

Ok...that demon thought is expelled now. I'm writing too many proposals. Approaching temporary burnout threshold. My brainstorming synapses may be totally misfiring at this point.


[Update 3 December 08:  2B searchable items is nary a drop in the bucket, I reckon.  Are we talking about 2 trillion?  200 trillion?  Large numbers that sound obscene when spoken?]