Integrated Concepts

This posting passed on by a colleague I find particularly interesting as it relates to our SOA implementation. The idea is that the four concepts:

Enterprise Canonical Data Model

Canonical Message Schema

Event Driven Architecture

Business Event Ontology

are intimately related and the post goes through the touch points for each. What I appreciate here is the separation of the canonical data model from the canonical message schema. In our implementation we create XSD to represent the business entities (analogous to the enterprise canonical data model) and then wrap various parts of these in XSD when creating the response, request, and acknowledgement message contracts (canonical message schemas).

But although we have a handle on these concepts I think it's important to really keep an eye on raising the visibility of the canonical data model from within the project team to the entire IT organizational level (thus making it canonical). Unfortunately we haven't been able to do this very successfully and so that limits the usefulness of the model both because we may not have considered what we need to and therefore the models may not be as robust as eventually needed, and since other groups may be off developing similar models that we don't know about. In order to address these limitations we're working with our architecture group to publish a "service catalog" that surfaces the schemas and their semantics along with what's been implemented thus far so that within the organization we'll be better able to communicate on a shared foundation. To do this we're contributing to a wiki that's been setup in Microsoft Office Sharepoint Server 2007.

In addition, while the messaging contracts are certainly useful they are not very flexible. In a second generation implementation we hope to move towards a more consumer-driven contract approach that retains the canonical data model schemas but allows consumers to pick and choose more easily the data they require.

On the fourth bullet we haven't been as forward-thinking as we would like. Although our implementation platform allows for the publication and consumption of business events we haven't integrated that publication into our infrastructure. Of course, as the poster mentions, first you need to figure out just what those events are and I'm hopeful that the idea of cataloguing those will gain some traction going forward.

MVP Summit Day 2

Day 2 of the MVP Summit was spent in building 43 on the Microsoft campus, the home of the Connected Systems Division. Right away what I liked about their approach to the event was that the sessions CSD delivered were not targeted to a single MVP focus (BizTalk for example) but rather to a larger group that included Solutions Architects and others. In essence the sessions were designed to give us the broad picture of how everything in the division interconnects, which, if you think about it, should be a primary focus of their division.

Just a quick note on last night's activities. The attendee dinner was held at The Museum of Flight south of downtown and it was a fabulous event. For some reason I had never made it to the museum and was absolutely floored by the quantity and quality of the exhibits. What was particularly interesting to me were the myriad examples of fighter planes from World War I up to the current day with brief but informative descriptions of each. The exhibits were also grouped by time period so a British Spitfire was next to the German Messerschmitt and the planes the US Navy flew were next to the Japanese versions which allowed you to compare and contrast them. The displays were also interesting in that along with a description of a specific kind of plane a brief blurb about the founder of the various aircraft companies as appropriate and the history of the museum's particular example of that aircraft.

In any case the first talk today was given by Sonu Arora, Program manager in CSD and Jesus Rodriguez, the Chief Software Architect of a company called Two Connect. Their topic was titled "Line of Business Adapter with WCF Deep Dive". Essentially the LOB Adapter is a new SDK for building adapters (termed the LOB adapter SDK where LOB refers to any backend application such as SAP, PeopleSoft and even custom databases) to integrate LOB applications with WCF clients. To that end the result is integration with your line of business applications that are…

Surfaced as WCF Binding

Can be consumed using WCF channel/service programming model

The strength of this approach is that it moves the adapter out of BizTalk and the other Microsoft products (for example Microsoft today has five SAP "adapters" strewn about their products) where it can be reused. In all, very interesting stuff.

The second session of the day was given by David Hill PM Tech Lead in the UI Framework regarding a topic that is still under NDA but is related to the existing client application block (CAB), Smart Client Software Factory, and WPF work being done and targeted for the post Visual Studio Orcas timeframe.

The third session was given by Jack Greenfield of Software Factories fame where he first discussed the current state of the software factory and software product lines and then moved on to the plans (under NDA) for this space under Rosario.

One of the things I did find interesting in his review of the current state of things is when he showed a slide that include a continuum of guidance evolution. The evolution starts with written guidelines, moves on to standard patterns, evolves to templates that can be reused in tools, then to frameworks that templates often rely on, and finally to visual designers that are integrated into the tool. This is interesting since in our project we're entering the template and frameworks stages by developing a set of CodeSmith templates to drive the development of new services by hooking into our existing service provider and service consumer frameworks. I'm not certain we'll ever get to visual designers or that we'll ever need to, but the topic reminded me that there is a continuum here and in my experience there is a large gulf between guidance/patterns and templates/frameworks and that is where you can really drive adoption. Until there is some actual help outside of the printed word, it is difficult for developers to adopt a new framework.

Next, we were off to building 5 and the home of the Patterns and Practices group where we took a tour of their agile-enabled redesigned space. In a nutshell the P&P group worked with a team of architects over a three to four month period and ended up rearchitecting their space for their teams that doing a mix of Scrum and Agile development.

The space, which accommodates up to 50 people, is now split into five team rooms wrapped by a ring of offices on a raised floor where literally the walls of all the team rooms and the offices are moveable. Each of the team rooms has desks with low dividers that can be configured in any way the team likes to the extent where the power coming from the floor can be moved along with the tables. The walls of each of the team rooms is double pane glass with a white material between them so they can be written on from floor to ceiling. Each of the team rooms was configured differently with some teams working in a circle sort of like a traditional conference room while others were staggered throughout the room. Not all of the walls had the white surface, however, and neither did the glass doors since one of the design criteria was that no matter where a team member sits, they can see outdoors. Each room also had a projector and conference phone and each phone at the workstations use IP and so can be moved as developers move around. The workstations themselves are all dual monitor (every team member uses a laptop) as well with their own lighting.

The outside offices are used for a variety of purposes ranging from a dedicated customer room where customers have a place to call home while working with the team to a series of offices some of which are actually dual offices for product managers and others. Interestingly, they also have a dedicated "quiet room" (my term) where a developer can get away from their team in order to make a phone call or otherwise take a break from their teammates. From the feedback I've gotten from the team I worked on who worked in a war room environment for several months, this is probably an essential aspect of this kind of team environment.

In taking the tour it was also noted that the P&P group of course uses Team Foundation Server but does not use MSBuild but rather CruiseControl to do their builds. They're also working their way through how to use the process guidance in TFS as the different teams have used different templates and use those different templates in various ways.

Developer Division Update

The Developer Division hour long session at the MVP summit was given by the corporate Vice President of the division by S. Somasegar. He started by presenting an overview of what has been happening within the division since the release of Visual Studio 2005 some 16 months ago. This included reviews of .NET 3.0 with WFC and WWF, ASP.NET Ajax, Team Foundation Server, the MSDN Wiki beta, and the March CTP of the Orcas release of Visual Studio. He then proceeded to bring out a presenter to do a demo of the Office 2007 integration with Orcas related to "forms regions". Essentially, as shown in the demo, within Orcas developers will have an add-in makes it fairly simple to integrate outside content into Outlook. The most interesting of the demos that he showed integrated additional features into contact management where clicking on a contact used Windows Live Services to pull down maps of the location within a pane inside outlook. He also showed a demo where business data on specific clients was brought up in a separate pane (using a WPF control highlighting the cool UI features) when an email from that client was selected. Interesting stuff.

Somasegar then went on to point out that "friction free" deployment was one of their big challenges and something they've been working hard on. He followed this with a demo of WPF/E (Windows Presentation Foundation Everywhere) (there was a CTP in February) and is the subset technology of WPF allowing rich multimedia experiences to be embedded on web pages using XAML and scripted using Javascript.

The demo was interesting in that it showed one of WPF/E enhanced capabilities to deal with inking. In the demo the user on the web was able to annotate an image or a video and then when posted back to the server (you could also use Ajax) and saved, the page would allow for searching the video stream for the annotation, seeking to it, and then replaying the inking on the screen.

He then answered a few questions and noted that PDC will be held this year in early October in Los Angeles.

At several points in the talk he was also very high on LINQ and urged those in attendance to take in the talk later today on the subject. It will also be included in Orcas and brings query capability down into the CLR and provides extensions to support both relational and XML data stores.

Global MVP Summit

After a brief visit to several spring training sites over the weekend I'm here in Seattle attending the Microsoft MVP Summit. This is the annual event where 1,700 Most Valuable Professionals from 90 plus countries gather to attend over 500 sessions and to give Microsoft feedback on their products. These MVPs (and I admit I'm woefully lacking in that area) do much of their work in newsgroups and provided over 28,000 answers (21% of all answers) to questions in the MSDN forums last year. I'm in the Visual Developer group as a Solutions Architect and will be attending many of the developer tracks and hopefully providing some commentary over the next several days.

In this morning's keynote Bill Gates spoke for about 20 minutes on the evolution of technology in Microsoft's just over 30 years in the industry. He hit all of the common themes from the growth in computing power via Moore's Law to the growth in multi-core computing (predicting 16 to 32 processors for desktop machines in 4 to 5 years) to the increasing availability of wireless technologies and it's expansion into the so-called "white space" that will allow for entire cities to provide wireless access. He also spent some time discussing graphics and video technologies and Microsoft's desire to raise the level of abstraction by providing APIs in the core operating systems that include the physics components and that also harness the power of the emerging multi-processor architectures. He sees video in many ways as the "final frontier" because of the challenges it posed in the past in terms of storage and processing power and he forsees the integration, cataloguing, and retrieval of video from a wide range of sources (colleges like MIT making their lectures freely available, businesses in corporate training, home and local use as in children's sporting events) as the next major application of the technology.

He also took question for over a half hour fielding a wide range of queries related to the history of Microsoft and its products, the decline of computer science graduates in the US, the role of information technology in disaster scenarios such as that posed by Katrina and 911 (he reference the show 24 in his answer to this one noting how the integration of systems that the CTU can tie into does not exist in the real world today), and the struggles of Windows Search.

Now it's on to the Developer Division Roadmap session.

Service Patterns and Data

I was alerted to this interesting article titled "Incorporating Enterprise Data into SOA" by a colleague and wanted to pass it along.

What I find interesting is that in our services implementation here we've built a mix of the first two patterns he discusses.

Aligning Enterprise Data Support with the Business Services. Here, our business services (meaning service operations) define and implement the rules for data validation and encapsulate the storing and retrieving particular sets of data albeit with the help of a set of "manager" classes that we use to decouple the data access logic from our EDAF implementation. This is most clearly evidenced in our service contracts by the fact that our "select" request schemas include the concept of "zones". Essentially, the long term data architecture at Compassion calls for data to be distributed in a series of zones ranging from OLTP to data warehouse cubes with the appropriate aging and flattening of the data as it flows through the zones.

To support this we created a schema that is a simple enumeration of zones and that can be included as an optional attribute in the envelope for service request schemas like so:


<xs:attribute ref="ds:DataStore" use="optional" />


Service consumers then set the zone accordingly and it is up to the service operation both to interpret the zone and retrieve the data from the appropriate location (perhaps even using different data access mechanisms). Clearly, here is a case where our business services know alot about our enterprise data.

Enterprise Data Access as a business service. Here some of the service operations we've built can be thought of as CRUD operations albeit with higher granularity thereby not falling into the "anti-pattern" category he mentions. For example, service operations such as LocateNeed which selects available children for sponsoring based on a set of rules and criteria, and QueryPaymentRelationshipSummary which summarizes and returns sets of monetary relationships between Compassion and our constituents are clearly well-aligned with business functionality.

That said, the tension between these two is what has led the schemas we developed to be in some cases too tightly bound to the existing enterprise data model. In a perfect world it would have been nice to work with a re-designed enterprise data model and to build a pure data services layer (enterprise data bus) on top of that which incorporates the concept of zones, validation, and entity aggregation that the more business oriented services would interact with.

Business Process Improvement

While thinking on the subject I thought I'd update the status of our project. We rolled out the final of our four releases (code named "Bread & Butter" and the most complex since it handled all types of money relationships including making payments on sponsorships, setting up new sponsorships with credit cards etc.) at the end of July and experienced a bit of a bumpy ride. After the rollout we were in a hotfix mode for roughly 30 days where we made incremental releases to various components every few days in order to handle issues that were not evidenced by our user acceptance testing. Unfortunately the release schedule was heavily influenced by some external business realities that the team didn't control and which forced us into the tradeoff situation we found ourselves in.

At this point however, the infrastructure has handled over 125,000 business transactions and 3.5M service invocations with system exceptions under 1%. The interesting thing, and the most important of course, is that 70% of the business transactions now go down the happy path and do not involve human interaction. That 30% that do require human interaction, however, are ripe for process improvements and additional incremental automation that will allow that number to shrink dramatically. There are two interesting aspects that came out of this however.

First, when you're implementing a business process management solution like this you should probably educate the business that there will be some low hanging fruit that comes out of the statistics generated after the release. As a result, they'll probably want to budget some resources for addressing these things in a final phase of the project. In our case, the project team is set to disband and the resources were not allocated and so the low hanging fruit will have to wait for another project. When that will exactly be is yet to be determined.

It's extrememly important to define before the project both the strategy for collection and the kinds of metrics you want to collect. The obvious reason is that these metrics will embody the questions that the business will ask and therefore point to where the process improvements can take place. In our implementation we did not do a thorough enough job of either collection or understanding and as a result there are some holes in our ability to ask the right questions. We're attempting to address some of this in a followup project but as you can imagine it's more difficult at this stage than it would have been if integrated into the early iterations of the project.

Fun in Omaha

Having a great time this morning at the Heartland Developer's Conference at the Qwest Center in Omaha Nebraska. Later today I'll be giving a session titled "Real SOA: Implementation and Best Practices" that discusses many of the topics found on this blog along with the experiences of the team over the almost 18 months since I came on board at Compassion as a Solutions Architect.

In short, the talk is broken into seven sections that highlight mostly what went right but also a little of what went wrong or things we could have done better. Those sections and the major point for each follow:

#1 Define a Services Platform
IT buy-in and support is critical

#2 Define Messaging and Schema Standards
It’s not just about coding standards

#3 Consider Cross Cutting Concerns
Build it for the long haul

#4 Consider Logging and Monitoring Up Front
Loosely coupled messaging is hard to tie together

#5 Build a Reference Model
Don’t jump in to business development

#6 Build a Reusable Framework
Budget time to help the rest of the organization

#7 Don’t forget business activity monitoring
Think about questions that will be asked at the end of the day

Always Learning

Last week we rolled out the last release in this year long project that developed and then proved out the service oriented infrastructure. Although we have issues to deal with related to the proper implementation of some of the uses cases (more on that in a later post) I thought I'd take a few minutes to discuss what I think is perhaps one of our biggest lessons learned.

In designing the contracts for our services we purposely wanted to introduce a layer of abstraction into the contracts so as to avoid creating service interfaces that were too closely tied to the existing enterprise data structure. Of course that's a balancing act since on the one had you need to have enough information represented in the various entity schemas to accurately identify the entity, and on the other hand not include extraneous attributes that serve to clutter the picture.

What we did well in this process is create the larger logical model where we introduced three high level concepts - Constituents, Needs, and Relationships. The first represents individuals of interest to Compassion, the second are entities whose needs can be satisfied such as a child to sponsor, a fund to donate to, or even an activity like praying for the work of the ministry. Relationships then represent the various connections between Constituents and Needs. For example, a Prayer Relationship is established between a Constituent and a Prayer Need while a Payment Relationship is established between a Constituent and a Child, Project, or Fund Need. This structure served us well because it allowed us to subdivide and represent each action for a constituent with a particular relationship type and more importantly allow our messaging infrastructure in BizTalk to recognize these relationship types and create and farm out work to independent orchestrations through a publish and subscribe model. This is an extensible model so that as new relationships are defined the infrastructure can handle them in a loosely coupled fashion (in fact, we implemented a new kind of action in this release without any modifications at all to the core BizTalk messaging model). So far so good.

The problem entered because going in we (and by that I mean I) didn't really understand how the logical model within the Payment Relationship should be defined. In other words, the Payment Relationship being the most complex, it needed a more robust definition at the outset in order to handle all of its variance. Unfortunately we rolled out the first definition of the Payment Relationship schema in our previous release when we were dealing only with relatively simple relationships that did not involve funded donations and commitments (in other words, we were dealing only with promises to pay and not persisting and handling rules for credit card and ACH transactions). And while there were place holders in the schema for this kind of information in the first version, the basic structure didn't extend as well as expected to handle this extended information. As a result, and here is the big idea, we ended up adding attributes to the schema that were too closely tied to the existing relational data model. To put it simply, too many codes right out of the database and not enough abstraction between the messaging model and the persistence layer. What happens when you do this of course is that you not only bring along the codes but their interrelationships and dependencies as well. Had we done more analysis up front, we would have seen these relationships and probably created higher level abstractions to handle them within our service contracts.

In one sense I'm glad we could roll out such a sophisticated system in a series of four iterations over a year, but in another the architectural tradeoffs that resulted from this approach (since we did not have the resources nor the time to fully flush this out ahead of time) will likely hamper us in ways as move down the road. But then again you're always living and learning and so we've toyed with the idea of version 2 service operations in some cases that provide these higher level abstractions.

Transitioning

For our project we're now about to release our final deliverable. This is the fourth deliverable in the project with the first one being in August of last year. This one is the largest and is used to automate many of the financial transactions performed on our web site which will in turn be used in the automation of requests that come from other channels (scanned forms and external call centers).

That said, the frameworks that the team has built are about to be used by other project teams as well. To that end one of the talks I attended at TechEd was a discussion of on assisting other developers in using framework that you build. One of the points made was that when you’re developing frameworks (like we have for services, orchestrations, and human workflow) the level of effort for framework developers doesn’t necessarily go down over time if you continue to refine it. The illustration used was the following graph.



As a result, the next effort for the team will be to transition from a project team doing business development into a team that continues to refactor and improve the frameworks that other developers use.

One of the other points made, however, was that teams should invest the time in the framework up front in order to make it consumable through the use of code generation, templates, etc. The presenter specifically mentioned that trying to put something together in a just in time fashion when other teams need it doesn’t work well in his experience. Unfortunately, because of the very tight schedule we've been on the past year that is largely where we are. The key insight for other teams doing this kind of work is to budget the time (I know how difficult that is when the work is viewed as "overhead" of course) in each release to refactor your frameworks and (reference architecture) and incorporate guidance as you go as well as allow time for mentoring other teams.