The Mr. Miyagi School of Software Engineering

This is a re-post from a journal entry I originally wrote over on Slashdot in 2004.  I've made some minor changes and updates for this entry.

Background:  This was an concept born out of necessity about 5 years ago.  I needed a way to train someone with very little experience to work with Bridgeworks but I couldn't afford to spend hours and hours of my own time working with him; too many of my own responsibilities and deliverables.  It was very successful and now he is our best Bridgeworks app developer.  Fast-forward to today when I'm interviewing CS students from a local university for a summer internship.  This is a (well) paid position for which I need someone with decent coding skills who is a self-starter.  Unfortunately none of my candidates have any practical skills whatsoever.  (I don't know what the universities are teaching, but it's not what we need in industry.)  With these candidates it would be like starting from square one.  Obviously I'm not going to pay what I'm willing to pay for people who don't have the quals.  I am willing to train such a person as an unpaid intern, but that is not without its catches.  I still don't have the time to spend hours and hours with these folks.  Furthermore, ethically, unpaid internships are intended for the sole benefit of the student.  My company should not benefit in any way from the labor of unpaid interns.  Practically speaking, though, you can't train someone without giving them tasks.  Re-enter the Mr. Miyagi School of Software Engineering.

Lessons on how to train a junior programmer:

Let's call him Danielsan.

Mr. Miyagi was a very wise and clever sensei. His methodology, loosely translated, is perfect for any small software company that is bringing new developers into the system. The reason Mr. Miyagi's method works so well is because it provides intense, immersive exposure to the most important lessons while demanding relatively few additional resources from the instructor(s). Think about it. While Danielsan was busy painting the fence and sanding the deck, Mr. Miyagi was out having the time of his life!

The length of each lesson is to be determined on a case by case basis.

Lesson 1: Write SDK Documentation

Even the best developers can be notorious for not adequately commenting their code. Good documentation of an software includes both programmer's notes and comments for automated documentation (e.g., doxygen). This oft neglected task is perfect for Danielsan. An excellent way for him to learn the software from a developer's perspective is to write the documentation that explains how it all works.

Listen carefully when Danielsan asks questions about the existing code base. Discourage him from asking too many questions, except regarding complex concepts. It is important that Danielsan develop his own understanding of algorithms, relationships, dependencies, etc.

Lesson 2: Build Company Demos

Documenting the code shows Danielsan how the software developer sees things. Danielsan also needs to see the software from users' perspectives. "Users" include programmers who develop applications from the software and end users of the applications that are developed. Ideally, these would be two separate lessons. Knowing that time and money are always issues, these lessons can be condensed into one by having Danielsan build the company demos.

Architects and senior engineers loathe building company demos. While they are often happy to write test apps for in-house use, company demos bring with them a mountain of maintenance headaches and customer support issues. Whether or not your company is big enough to have it's own department(s) for maintenance and support, it's worth putting Danielsan to work on company demos so he can get his hands a little dirty and see first hand the challenges facing maintenance and support team(s).

Lesson 3: Clean House

Many companies have coding standards that must be followed by all code writers. These standards help everyone to write clean, consistent code that everyone can understand. Unless you work at a sterlized laboratory, it's a safe bet that your house always needs cleaning. A great way for Danielsan to learn this important lesson and also develop habits that are consistent with the team is to set him to work checking for adherence to company coding standards, leaks, potential security issues and the like. It's also a convenient way for you to get your code checked by a fresh pair of eyes.

The Successful Sensei

The successful sensei will know that these lessons are not opportunities for him to relax his own standards or to set Danielsan to work unguided. The successful sensei practices what he preaches. He knows which lessons Danielson must learn on his own and which require guidance. Naturally, Danielsan may occasionally bemoan his instruction. Perhaps he will consider that his training is too rote or mundane. Since you can't just throw a bunch of karate maneuvers in his face to show him what he's learned, it's important to sometimes let Danielsan work on things he finds interesting and fun. Ask him to write stand-alone utility apps that your company might need. Give Danielsan isolated new tasks within the SDK, perhaps something that requires he work with others to design interface requirements, resource requirements, etc.

to be cont'd.

Axiom 1: Tooltips are better left on.

If you leave your tooltips on, chances are better that you will learn something new each time you use your application(s). For Danielsan, tooltips are especially useful when they instruct him about fundamental principles of programming, those that transcend applications.


Axiom 1a: Some tooltips are better than others.

On Ambient Visualization

I want visualization to be less a part of a specific application that I go to and to be more of a natural extension to the computer itself, available from everywhere.  I want visualization to an ambient experience.  When I encounter any table of data in any document container I'd like to be able to quickly view it as a column chart without starting up a chart-making or data processing application, without shuffling around through copy & paste.  I just want to select rows and columns an pop a "window" with a chart in it in one easy step.  If I can recognize with my eyes that a table contains place names or lat/lon pairs then a computer ought to be able to map it with minimal intervention on the part of the user.  I should also be able to put my selected, obviously geospatial data on a virtual Earth model.  With just a little more imagination I can see turning lists and tables into nodes and edges, viewable with graph layouts.  Think Enso for visualization.  I might want to do more than just look at my chart, map/Earth model, and graph.  I might want to start to interact with these views (assume independently for now).  It starts to seem like I need an application to do that, but I'm not ready to jump the gun.  This is still in the realm of a capability and not necessarily an application.  Applications start to assume containers and domain-specific use cases.  Most visualization techniques have standard, "off-the-shelf" things you can do with them given basic commands or input devices.  Charts can be sorted and transformed into different layouts.  One can pan, zoom, and rotate maps and terrain.  The technique of "drill-down" and "roll-up", which can be applied to any visualization technique, is nothing more than navigation of linked data at multiple levels of detail and sometimes across multiple view contexts.  At what point do is a specialized application needed more than a capability?  We may be overly conditioned to assume the application model when we think of software as having utility.  This is changing rapidly on the web.  (It was always thus on the Unix command line, yes?)  Visualization ought to change with it.  Leave the application building up to subject matter experts with an application domain, not to software programmers.  Ah but wait, lest too much be read into a passing editorial remark.  Obviously software programmers play a key role here.  The tendency among programmers who attempt to answer that call is to build an "application building framework."  Again, the assumption is that subject-matter experts always need an app to make use of viz.  I wonder why visualization software shouldn't be a part of an operating system; a core capability for any application or purpose.  There have been wonderful advances made through web browser extensions, but even here visualization is at best an after thought applied to a mostly universal application.  (I say "mostly" b/c there are no less than 3 different web browsers installed on my one operating system.)  What happens when someone emails me some data in a flat file that I open in a text editor?    Instead what is need are document object models for visualization techniques and runtime software that can parse viz documents on the fly.  The runtime is optimized for robust interaction and attribute manipulation of high level visual artifacts, not application-specific tasks.  This runtime can be invoked from a background process or "embedded" (contained in, called from) an application runtime.  Devices having different display and user interfaces can choose how to represent what are otherwise well understood visual metaphors.  Data can be more easily passed around and visualized simply by passing text documents describing interactive, dynamically updatable (or not) views.  (This seems inherently more secure, too.)  Only this way is ambient visualization possible; something that is available everywhere on my computer, no matter what kind of computer/device/hardware platform I am using.

Carl Malamud for Public Printer of the United States

I endorse Carl Malamud for Public Printer of the United States

(Administrator of the US GPO)

Enterprise Architectures As Data 2

I wrote about this topic last December.  Now I want to drill down on a key element of our approach for turning enterprise architecture models into queryable sources.  Furthermore,  I should address why anyone would ever care about "turning enterprise architecture...blah blah blah."

There is a technology aspect and a human aspect.  Both are necessary, but technology alone is never sufficient.  Far more important to our success has been a strategy focused on answering key stakeholder questions.  

My experience is mostly with Department of Defense Architecture Framework (DoDAF), so I will use examples from that domain.  But to the best of my ability I will use language that is portable to other domains.

So, what are "key stakeholder questions?"  To what does "key" refer, "stakeholder" or "questions?"  The answer to the second question is:  both.  The answer to the first question is:  depends on which stakeholder(s) we are talking about.

In the Army, and DoD in general, organizations are broken down at the general officer level from 1-9 as follows:  

1. HR
2. Intel
3. Operations
4. Logistics/Sustainment
5. Planning/Doctrine
   
6. IT
7. Training
8. Finance
9. Civil-Military Cooperation

These are the key stakeholder perspectives at the highest level.  

An Operator wants to know, "How will architecture help me win the fight (i.e., get the work done)?"  A Budget Analyst wants to know, "How does architecture help me making smarter spending decisions?"  These are reasonable questions.  

Indeed, no model exists except to answer questions.  

But when presented with architecture in forms that speak only to the IT/engineering perspective, i.e., schematics, flow diagrams and the like, the multi-stakeholder conclusion tends to be, "Architecture doesn't help answer my questions."  That's unfortunate because architecture, especially at the scale and complexity of federal government, can be effective in answering many useful questions for all perspectives.  The trick is to:

1. Understand what questions are being asked from each perspective
   
2. Extend architectures from models that produce schematics into queryable sources that can produce schematics and also answer questions
3. Extract the answers to those questions automatically to an open, text-based format
4. Transform answers into views that are "fit for purpose," i.e., can be understood and used from many perspectives

Therefore, the question IT people should be asking is, "What can I do to make architecture better able to answer stakeholder questions."  To be sure, there are questions architectures can't answer.  But it is more productive to focus on the questions that architecture can answer.  Following are actual endpoints in use (though they don't point anywhere in this post):

GetCountOfPlatformsForArch
Returns an Xml Node containing a NewDataSet for the distribution of platforms for the given architecture
Answers the question: What are the platforms of this architecture and how many of each?

GetCountOfPlatformsForArchAndTOE
Returns an Xml Node containing a NewDataSet for the distribution of platforms for the given architecture and TOE
Answers the question: What are the platforms of this architecture and TOE and how many of each?

GetEquipmentForONN

Returns an Xml Node describing the equipment used by the given ONN in the given architecture
Answers the question: What equipment does this ONN use in this architecture? 

Believe it or not, this type of view can be automatically produced 

from an enterprise architecture model

There is added value in this approach, too.  From the technology side of the coin, opening up architecture models through relational databases and ubiquitous, text-based formats makes it much easier to relate architecture data with other data.  An architecture can tell you "how many," but "how much" is probably in another database.  In fact, there are about 85 software programs the Army knows about that are being used to track costs for systems.  To be sure some will be eliminated.  But many will stay and those that do must support data portability.

Ok, so...why does any of this matter?

Let us start with the fact that government is an enormous compiler of data.  It really doesn't matter if we are talking about enterprise architecture, a Line Item Number database, a "portfolio management" system, or even good, old-fashioned spreadsheet.  Analysts will never part with their spreadsheets.  Wonks stuff data into tools.

Unfortunately, many useful tools that are good at capturing data are lousy at both storing it and saving it back out.  Applications too often co-mingle their own data with the subject matter data, making it difficult to separate later.  Many tools don't have export in mind.  (All tools should have export in mind.)  Owners of online, enterprise databases typically only offer their contents in human-readable formats; PDF and HTML pages. Applications built for one perspective are lousy at supporting other perspectives.  The list of interoperability challenges goes on.

A battalion commander is not typically nimble with UML.  Same goes for a budget analyst and resource director.  In the world of enterprise architecture the consequence is that a lot of people are taking a lot of time away from their primary tasks so that they can manually shuffle data into views their many customers demand.  Time is money.  My back of the napkin estimate is that architecture shops are working twice as hard for half the output that they could otherwise achieve if they offloaded the ETLV problem to IT and got back to building architectures.  Of course, plenty will point out that architecture and reality are often different.  That's a topic for another discussion.  Like it or not, enterprise architecture plays a huge role in government organizations, especially DoD.  This is just about getting better return on that investment.  It's opportunity for both a current cost savings and future "force multiplier."

Finally, I would argue that the strategy of answering key stakeholder questions is useful to knowledge management in general, regardless of subject matter domain or technology form factor.

"Response" to Previous

Update 12.28.09:  I just had to record for posterity the flames from some folks who tagged my previous post in Reddit.  (So much for polite, constructive discourse.)  In truth, I would like to have this conversation with some of these folks.  Even the harshest ones.  But I don't want to have to register for a service I don't use and I don't want to get into flame wars over comment threads.  I get the frustration.  I've experienced it.  I too have been and remain rather skeptical of the hype.  Some level of hype gets things noticed and forces conversations that need to happen but aren't.  Some of what I said was written and/or read poorly.  For example, Mac/Office is about culture, not interoperability.  I know that Google came before Web 2.0.   That was expressed poorly.  My company also was doing a lot of this stuff before we ever heard of Web 2.0.  I can't speak for Tim O'Reilly, but I feel confident he realizes he didn't invent something.  He made salient observations about the Internet and human behavior and how the two have, do, and perhaps ought to work together.  Many of these concepts go back decades to the roots of computer science and the Internet.  I've been in workshops with Senior Technical Fellows, having extremely well-qualified CVs, who said, "we proposed all this in the beginning."  My response was, "Exactly.  Isn't it past time we got back to that?" 

And I know how radical ("insane") my idea sounds, but I am not saying government should fund Web 2.0.  This is not about a bailout, as some seem to think.  Some of these comments come from the position of having no idea how government runs.  I'm saying government is actively spending money, money is being wasted, and I know from education and direct, relevant experience that several of the concepts articulated in the original essay can help tax paying citizens spend more wisely.

Consider this from the Army:

"The Army spends, under 85 programs, approximately $6.7B annually on Information Technology (I.T.) without a method to converge these systems into a centralized infrastructure designed to improve robustness and dynamically deliver web services to hundreds of thousands of users while reducing risk"

And that is just service within DoD, DoD just being one (the biggest) government spender on IT.  It's full of waste and the reason isn't just because Web 2.0 is magic Koolaid or that we are already on to so-called Web 3.0.  It is because the government still looks at the Web the way it used to be in the 90s; i.e., with a 1.0 mindset.

More concrete thoughts on this later on how exactly government might use Twitter, for example.  Think SMS.  Twitter is  a metaphor.  The folks who run Twitter - for example - just might be the Pros from Dover who can help save government from itself.  Think FEMA and USAID...

Here's the juicy stuff:

Haha, this nutbag thinks that the government should fund web 2.0. And he's serious.(kevincurry.blogspot.com)

submitted 4 days ago by candlejac

• 11 comments
• share

sort by: 

hot

Kitchenfire 3 points 4 days ago[-]

""The Bigs," i.e., large-cap companies that provide most of the contracting labor, are not at all oriented to innovate in the Web 2.0 technology space. You don't see Macs anywhere. You do see MS Office everywhere. "

I wonder if this guy knows that Macs can run MS Office. Or that he's insane.

• permalink
• reply

turkourjurbs 3 points 4 days ago[-]

"Google does a good job selling into government with its enterprise appliance model, and with more than just search. But, of course, Google is a massive company."

SIGH!!! Google was aroung long before Tim O'Reilly decided he needed more money, and made up a completely inaccurate and bullshit term to describe something we already have. Even if he didn't decide to look like a technological ignoramus, we'd still have Twitter, Facebook, etc. without calling them something that makes absolutely no sense at all.

Please, point out which "Web 2.0 Server" I should be using and which "Web 2.0" browsers will work with them. Every underlying technology that's considered "web 2.0" is the same technology that's been behind the web since before there was Web 2.0 We have the web. We have web sites. There is nothing more to it.

Maybe if we somehow get rid of the head nutbag (O'Reilly), the rest of the web 2.0 delusionists like this one will go with him.

• permalink
• reply

skymt0 2 points 4 days ago[-]

Please, point out which "Web 2.0 Server" I should be using

That would be lighttpd, according to their home page.

• permalink
• parent
• reply

cochico 1 point 4 days ago[-]

pffft! We're already working on Web 3.0

• permalink
• reply

funkah 1 point 4 days ago[-]

Crazy, sure, but I'm not exactly loving the idea of giving tons of money to banks who lost hundreds of billions because of shitty risk analysis, either.

• permalink
• reply

candlejac 1 point 4 days ago[-]

At least their risk analysis is better than Twitter's.

• permalink
• parent
• reply

funkah 2 points 4 days ago[-]

I don't even know what that means.

• permalink
• parent
• reply

candlejac 1 point 4 days ago* [-]

Twitter is a company with no business plan. They seriously are closest to the underpants gnomes

1. Build website for 140 character microblog posts
2. Pay carriers to send txts with updates to subscribers, while not charging for this service
3. Pay to receive texts on a shortcode
4. ???
5. Profit!

• permalink
• parent
• reply

grilled_ch33z 1 point 4 days ago* [-]

I'd argue that it's more like:

• Build website for 140 character microblog posts
• Pay carriers to send txts with updates to subscribers, while not charging for this service
• Pay to receive texts on a shortcode
• ???
• ???

edit: how do you do numbered lists?

• permalink
• parent
• reply

candlejac 2 points 4 days ago[-]

numbers and periods, but you must not know much about the underpants gnomes

• permalink
• parent
• reply

Ac3 2 points 4 days ago[-]

Well he clearly will not profit.

• permalink
• parent
• reply

Q: How Does Web 2.0 Make Money? A: Government.

A lot of folks are wondering how Twitter will monetize.  Will they sell premium services to businesses that want to make Twitter part of a communications strategy?  What about Flickr, YouTube, and Facebook?  Are the ads working?  Regardless, if this is Web 2.0 then why are we still talking about subscriptions and eyeballs on pages?! 

I want to suggest another strategy:  Sell to government.  

I don't mean the general sense.  I  mean, specifically, Twitter, YouTube, and Facebook and a great many others should set up for-government operations.  I wrote to the Delicious team years ago asking when they were coming out with a solution that I could bring to government, i.e., in-house.  I never got an answer.  Regardless, figuring out how to get these companies oriented toward government is not straightforward.  Most Web 2.0 companies literally could not be further outside The Beltway.  I suspect they don't have much in the way of strategy for state, tribal, or local either.  Google does a good job selling into government with its enterprise appliance model, and with more than just search.  But, of course, Google is a massive company.

Culture has a lot to do with things.  The Pentagon is not a T-shirt and flip flops kind of environment.  "The Bigs," i.e., large-cap companies that provide most of the contracting labor, are not at all oriented to innovate in the Web 2.0 technology space.  You don't see Macs anywhere.  You do see MS Office everywhere.  I'm not entirely sure what conclusions can be drawn from these observations but I am sure the observations are significant.

I suppose the best example of Web 2.0 penetration into the government space is tele-presence.  Adobe Breeze is ubiquitous on Defense Knowledge Online (DKO).  Just about anyone with a DKO government account can create or attend a meeting.  But tele-presence probably isn't the first thing that comes to people's minds when asked to name a Web 2.0 technology and I'm not sure Adobe is the best example of a Web 2.0 company.

Yes, there is something different about Twitter, Flickr, YouTube, Facebook, Delicious, et al.  I happen to think that something different - whatever it is - translates into unrealized opportunity for both buyers and sellers in the government space.  I choose to focus on these technologies specifically and there are others that I include.  ProgrammableWeb has a solution for the registry problem, for example.  I don't represent any of these companies, by the way.  I call them out as (mostly) well-known examples of capabilities the government needs.  I don't really care if Flickr, Picasa, or PhotoBucket is the image repository of choice.  Vimeo and YouTube can and should compete for the video infrastructure.  

The point is that government needs platform solutions.

Simple, content-based, platform solutions are the most obvious plays for Web 2.0 in government; images, video, audio.  The federal government processes a staggering amount of this stuff.  The DoD may be the first to get into the mix with TroopTube. 

There also are plenty of outfits that would use social tagging tools if only they could bring them in house.  By bring them in house, what I mean and recommend is providing enterprise services solutions.  It's nice to have applications that provide tagging, but applications are seldom the best enterprise solutions and are hardly social (except, perhaps, in MOSS).  Tagging is a domain, a Web 2.0 partition, if you will, unto itself.  It is a simple-enough-but-not-too-simple utility that scales and it can be integrated with just about any other application, regardless of whether or not the application was designed with tagging in mind.  Yes, government needs strategic guidance and support for tagging services and Web 2.0 tagging companies are just the ones to provide it...if we can figure out how.

The current providers, a.k.a. "The Bigs" are not oriented to provide Web 2.0 tech support.  

This is either an opportunity to get new business before The Bigs or to create a business of showing them the way.  In any case it is an opportunity to make money.  More specifically, it is a way for Web 2.0 technologies to make money.  In the government services space the biz-speak is generally referred to as "priming" and "subbing," as in:  you are either a prime contractor or a subcontractor.  I don't see many Web 2.0 companies subcontracting to big, traditional system integrators, though.

The government is not set up to acquire Web 2.0 technologies.  

While services contracts are fairly common and understood, things are less well-defined in the products and solutions space unless the products and solutions are ubiquitous or otherwise extremely well known.  Government, especially federal, wants to buy everything in bulk.  See also:  Indefinite Delivery, Indefinite Quantity (IDIQ).  New technology is extraordinarily difficult to insert, especially in secure environments.  Protocol, procedure, and process rule the land.  Force-fitting into an existing model is too often the preferred method.

We are getting better at defining and avoiding undue process but process, by definition, is:  1)  necessary and 2)  inherently situational.

When doing business with the federal government, it's important to know how business is done inside The Beltway.  There are most probably things in the mix that need to be undone, too.  To a meaningful extent the situation is not different at state, tribal, and local levels.  Education is needed on both sides of the buyer/seller relationship.  We might "change the world" in the process of implementing Web 2.0 for government, but requisite is the obligation to have a fair understanding of "the world" first.  The obligation goes both ways but mostly falls on the shoulders of sellers.

Choices for Web 2.0 companies to make money by doing business with government:

1. Become or spin-off an enterprise systems integrations unit
2. Sell enterprise systems solutions a) to government b) to system integrators (Note:  probably can't sell solutions to government without either being an integrator or having the support of one.)
3. Consult a) to government b) to system integrators on enterprise systems

Of course, "partner" is an option, but still implies one or more the previously listed options.

[Update 9:47 PM - I've decided I really don't like these choices at all.  Need to come up with an altogether new business model, perhaps...probably]

There's so much more to this than I can wrap my head around now, certainly more than I am prepared or qualified to comment here.

[Update 9:47 PM - Forgot to comment on need for and evidence of government investment in backbone instracture and understanding of cloud architecture; significant issues arise once a bunch of these services are running around on a single network.  And as always, security is different and harder.]

I think it's time Web 2.0 companies, government, and large-cap contracting companies had a grand introduction to one another.  Believe it or not, there are plenty of people who have never even heard of Web 2.0.

Common Web 2.0 Services for Government Brainstorm

1. SMS - ex., Twitter
2. Expertise Location - ex., Facebook, LinkedIn, semantic profilers
3. Tagging - ex., Delicious, Magnolia, Stumble Upon
4. Image Sharing- ex., Flickr, Picasa, PhotoBucket
5. Video Sharing- ex., YouTube, Vimeo, TroopTube! (12.28.08)
6. Audio Sharing - ex., SoundCloud, HuffDuff
7. Document Sharing - ex., Google Docs, MOSS, Word Press, Blogger, Tumblr
8. Registry - ex., ACME.ProgrammableWeb, Wiki (?)
9. Tele-presence - ex., Breeze, WebEx, GoToMeeting
10. Search - ex. Google, Yahoo!
11. Visualization - ex., ManyEyes, Swivel
12. Dictionary - ex., Merriam-Webster.com (http://m-w.com/dictionary/[WORD])
13. Data Transformation- ex., ??
14. [Update 12.29.08] Geospatial, mapping - ex., Google Earth/Maps, Microsoft Live Virtual Earth/Maps

• Utility model (i.e., like electric, water, natural gas)
• Specialized content and application services infrastructure
• Government needs a strategy for inserting these technologies
• Government strategy must be capabilities-based and vendor indifferent, yet cannot be generic
• How exactly/specifically can government do business with Facebook or LinkedIn, for example?
• To what extent will/should third-party integrators be involved?  Is Twitter likely to provide labor resources for technology insertion or would they just want to license the platform to third parties?
• Companies listed above need a strategy for doing business with government.  Some have them, but most don't.
• The mutual strategy should be for companies to implement their architectures for these platform services in public and government-managed domains.
• Probably the biggest hurdle is the massive amount of process and procedure required to navigate the government marketplace and interact with customers
• Culture gaps

More later...

[Update 12.23.08]

• Security model is much more complicated than just protecting access to personal information; ease of and tracking of information flow a greater concern
• Security models need to be reconsidered by both buyers and sellers