Making SOA Real

Integrated Concepts

2007-06-21T07:12:00.001-07:00

This posting passed on by a colleague I find particularly interesting as it relates to our SOA implementation. The idea is that the four concepts:

Enterprise Canonical Data Model

Canonical Message Schema

Event Driven Architecture

Business Event Ontology

are intimately related and the post goes through the touch points for each. What I appreciate here is the separation of the canonical data model from the canonical message schema. In our implementation we create XSD to represent the business entities (analogous to the enterprise canonical data model) and then wrap various parts of these in XSD when creating the response, request, and acknowledgement message contracts (canonical message schemas).

But although we have a handle on these concepts I think it's important to really keep an eye on raising the visibility of the canonical data model from within the project team to the entire IT organizational level (thus making it canonical). Unfortunately we haven't been able to do this very successfully and so that limits the usefulness of the model both because we may not have considered what we need to and therefore the models may not be as robust as eventually needed, and since other groups may be off developing similar models that we don't know about. In order to address these limitations we're working with our architecture group to publish a "service catalog" that surfaces the schemas and their semantics along with what's been implemented thus far so that within the organization we'll be better able to communicate on a shared foundation. To do this we're contributing to a wiki that's been setup in Microsoft Office Sharepoint Server 2007.

In addition, while the messaging contracts are certainly useful they are not very flexible. In a second generation implementation we hope to move towards a more consumer-driven contract approach that retains the canonical data model schemas but allows consumers to pick and choose more easily the data they require.

On the fourth bullet we haven't been as forward-thinking as we would like. Although our implementation platform allows for the publication and consumption of business events we haven't integrated that publication into our infrastructure. Of course, as the poster mentions, first you need to figure out just what those events are and I'm hopeful that the idea of cataloguing those will gain some traction going forward.

MVP Summit Day 2

2007-03-14T16:27:00.000-07:00

Day 2 of the MVP Summit was spent in building 43 on the Microsoft campus, the home of the Connected Systems Division. Right away what I liked about their approach to the event was that the sessions CSD delivered were not targeted to a single MVP focus (BizTalk for example) but rather to a larger group that included Solutions Architects and others. In essence the sessions were designed to give us the broad picture of how everything in the division interconnects, which, if you think about it, should be a primary focus of their division.

Just a quick note on last night's activities. The attendee dinner was held at The Museum of Flight south of downtown and it was a fabulous event. For some reason I had never made it to the museum and was absolutely floored by the quantity and quality of the exhibits. What was particularly interesting to me were the myriad examples of fighter planes from World War I up to the current day with brief but informative descriptions of each. The exhibits were also grouped by time period so a British Spitfire was next to the German Messerschmitt and the planes the US Navy flew were next to the Japanese versions which allowed you to compare and contrast them. The displays were also interesting in that along with a description of a specific kind of plane a brief blurb about the founder of the various aircraft companies as appropriate and the history of the museum's particular example of that aircraft.

In any case the first talk today was given by Sonu Arora, Program manager in CSD and Jesus Rodriguez, the Chief Software Architect of a company called Two Connect. Their topic was titled "Line of Business Adapter with WCF Deep Dive". Essentially the LOB Adapter is a new SDK for building adapters (termed the LOB adapter SDK where LOB refers to any backend application such as SAP, PeopleSoft and even custom databases) to integrate LOB applications with WCF clients. To that end the result is integration with your line of business applications that are…

Surfaced as WCF Binding

Can be consumed using WCF channel/service programming model

The strength of this approach is that it moves the adapter out of BizTalk and the other Microsoft products (for example Microsoft today has five SAP "adapters" strewn about their products) where it can be reused. In all, very interesting stuff.

The second session of the day was given by David Hill PM Tech Lead in the UI Framework regarding a topic that is still under NDA but is related to the existing client application block (CAB), Smart Client Software Factory, and WPF work being done and targeted for the post Visual Studio Orcas timeframe.

The third session was given by Jack Greenfield of Software Factories fame where he first discussed the current state of the software factory and software product lines and then moved on to the plans (under NDA) for this space under Rosario.

One of the things I did find interesting in his review of the current state of things is when he showed a slide that include a continuum of guidance evolution. The evolution starts with written guidelines, moves on to standard patterns, evolves to templates that can be reused in tools, then to frameworks that templates often rely on, and finally to visual designers that are integrated into the tool. This is interesting since in our project we're entering the template and frameworks stages by developing a set of CodeSmith templates to drive the development of new services by hooking into our existing service provider and service consumer frameworks. I'm not certain we'll ever get to visual designers or that we'll ever need to, but the topic reminded me that there is a continuum here and in my experience there is a large gulf between guidance/patterns and templates/frameworks and that is where you can really drive adoption. Until there is some actual help outside of the printed word, it is difficult for developers to adopt a new framework.

Next, we were off to building 5 and the home of the Patterns and Practices group where we took a tour of their agile-enabled redesigned space. In a nutshell the P&P group worked with a team of architects over a three to four month period and ended up rearchitecting their space for their teams that doing a mix of Scrum and Agile development.

The space, which accommodates up to 50 people, is now split into five team rooms wrapped by a ring of offices on a raised floor where literally the walls of all the team rooms and the offices are moveable. Each of the team rooms has desks with low dividers that can be configured in any way the team likes to the extent where the power coming from the floor can be moved along with the tables. The walls of each of the team rooms is double pane glass with a white material between them so they can be written on from floor to ceiling. Each of the team rooms was configured differently with some teams working in a circle sort of like a traditional conference room while others were staggered throughout the room. Not all of the walls had the white surface, however, and neither did the glass doors since one of the design criteria was that no matter where a team member sits, they can see outdoors. Each room also had a projector and conference phone and each phone at the workstations use IP and so can be moved as developers move around. The workstations themselves are all dual monitor (every team member uses a laptop) as well with their own lighting.

The outside offices are used for a variety of purposes ranging from a dedicated customer room where customers have a place to call home while working with the team to a series of offices some of which are actually dual offices for product managers and others. Interestingly, they also have a dedicated "quiet room" (my term) where a developer can get away from their team in order to make a phone call or otherwise take a break from their teammates. From the feedback I've gotten from the team I worked on who worked in a war room environment for several months, this is probably an essential aspect of this kind of team environment.

In taking the tour it was also noted that the P&P group of course uses Team Foundation Server but does not use MSBuild but rather CruiseControl to do their builds. They're also working their way through how to use the process guidance in TFS as the different teams have used different templates and use those different templates in various ways.

Developer Division Update

2007-03-13T13:16:00.000-07:00

The Developer Division hour long session at the MVP summit was given by the corporate Vice President of the division by S. Somasegar. He started by presenting an overview of what has been happening within the division since the release of Visual Studio 2005 some 16 months ago. This included reviews of .NET 3.0 with WFC and WWF, ASP.NET Ajax, Team Foundation Server, the MSDN Wiki beta, and the March CTP of the Orcas release of Visual Studio. He then proceeded to bring out a presenter to do a demo of the Office 2007 integration with Orcas related to "forms regions". Essentially, as shown in the demo, within Orcas developers will have an add-in makes it fairly simple to integrate outside content into Outlook. The most interesting of the demos that he showed integrated additional features into contact management where clicking on a contact used Windows Live Services to pull down maps of the location within a pane inside outlook. He also showed a demo where business data on specific clients was brought up in a separate pane (using a WPF control highlighting the cool UI features) when an email from that client was selected. Interesting stuff.

Somasegar then went on to point out that "friction free" deployment was one of their big challenges and something they've been working hard on. He followed this with a demo of WPF/E (Windows Presentation Foundation Everywhere) (there was a CTP in February) and is the subset technology of WPF allowing rich multimedia experiences to be embedded on web pages using XAML and scripted using Javascript.

The demo was interesting in that it showed one of WPF/E enhanced capabilities to deal with inking. In the demo the user on the web was able to annotate an image or a video and then when posted back to the server (you could also use Ajax) and saved, the page would allow for searching the video stream for the annotation, seeking to it, and then replaying the inking on the screen.

He then answered a few questions and noted that PDC will be held this year in early October in Los Angeles.

At several points in the talk he was also very high on LINQ and urged those in attendance to take in the talk later today on the subject. It will also be included in Orcas and brings query capability down into the CLR and provides extensions to support both relational and XML data stores.

Global MVP Summit

2007-03-13T11:25:00.000-07:00

After a brief visit to several spring training sites over the weekend I'm here in Seattle attending the Microsoft MVP Summit. This is the annual event where 1,700 Most Valuable Professionals from 90 plus countries gather to attend over 500 sessions and to give Microsoft feedback on their products. These MVPs (and I admit I'm woefully lacking in that area) do much of their work in newsgroups and provided over 28,000 answers (21% of all answers) to questions in the MSDN forums last year. I'm in the Visual Developer group as a Solutions Architect and will be attending many of the developer tracks and hopefully providing some commentary over the next several days.

In this morning's keynote Bill Gates spoke for about 20 minutes on the evolution of technology in Microsoft's just over 30 years in the industry. He hit all of the common themes from the growth in computing power via Moore's Law to the growth in multi-core computing (predicting 16 to 32 processors for desktop machines in 4 to 5 years) to the increasing availability of wireless technologies and it's expansion into the so-called "white space" that will allow for entire cities to provide wireless access. He also spent some time discussing graphics and video technologies and Microsoft's desire to raise the level of abstraction by providing APIs in the core operating systems that include the physics components and that also harness the power of the emerging multi-processor architectures. He sees video in many ways as the "final frontier" because of the challenges it posed in the past in terms of storage and processing power and he forsees the integration, cataloguing, and retrieval of video from a wide range of sources (colleges like MIT making their lectures freely available, businesses in corporate training, home and local use as in children's sporting events) as the next major application of the technology.

He also took question for over a half hour fielding a wide range of queries related to the history of Microsoft and its products, the decline of computer science graduates in the US, the role of information technology in disaster scenarios such as that posed by Katrina and 911 (he reference the show 24 in his answer to this one noting how the integration of systems that the CTU can tie into does not exist in the real world today), and the struggles of Windows Search.

Now it's on to the Developer Division Roadmap session.

Service Patterns and Data

2006-11-28T11:11:00.000-08:00

I was alerted to this interesting article titled "Incorporating Enterprise Data into SOA" by a colleague and wanted to pass it along.

What I find interesting is that in our services implementation here we've built a mix of the first two patterns he discusses.

Aligning Enterprise Data Support with the Business Services. Here, our business services (meaning service operations) define and implement the rules for data validation and encapsulate the storing and retrieving particular sets of data albeit with the help of a set of "manager" classes that we use to decouple the data access logic from our EDAF implementation. This is most clearly evidenced in our service contracts by the fact that our "select" request schemas include the concept of "zones". Essentially, the long term data architecture at Compassion calls for data to be distributed in a series of zones ranging from OLTP to data warehouse cubes with the appropriate aging and flattening of the data as it flows through the zones.

To support this we created a schema that is a simple enumeration of zones and that can be included as an optional attribute in the envelope for service request schemas like so:


<xs:attribute ref="ds:DataStore" use="optional" />

Service consumers then set the zone accordingly and it is up to the service operation both to interpret the zone and retrieve the data from the appropriate location (perhaps even using different data access mechanisms). Clearly, here is a case where our business services know alot about our enterprise data.

Enterprise Data Access as a business service. Here some of the service operations we've built can be thought of as CRUD operations albeit with higher granularity thereby not falling into the "anti-pattern" category he mentions. For example, service operations such as LocateNeed which selects available children for sponsoring based on a set of rules and criteria, and QueryPaymentRelationshipSummary which summarizes and returns sets of monetary relationships between Compassion and our constituents are clearly well-aligned with business functionality.

That said, the tension between these two is what has led the schemas we developed to be in some cases too tightly bound to the existing enterprise data model. In a perfect world it would have been nice to work with a re-designed enterprise data model and to build a pure data services layer (enterprise data bus) on top of that which incorporates the concept of zones, validation, and entity aggregation that the more business oriented services would interact with.

Business Process Improvement

2006-10-26T08:18:00.000-07:00

While thinking on the subject I thought I'd update the status of our project. We rolled out the final of our four releases (code named "Bread & Butter" and the most complex since it handled all types of money relationships including making payments on sponsorships, setting up new sponsorships with credit cards etc.) at the end of July and experienced a bit of a bumpy ride. After the rollout we were in a hotfix mode for roughly 30 days where we made incremental releases to various components every few days in order to handle issues that were not evidenced by our user acceptance testing. Unfortunately the release schedule was heavily influenced by some external business realities that the team didn't control and which forced us into the tradeoff situation we found ourselves in.

At this point however, the infrastructure has handled over 125,000 business transactions and 3.5M service invocations with system exceptions under 1%. The interesting thing, and the most important of course, is that 70% of the business transactions now go down the happy path and do not involve human interaction. That 30% that do require human interaction, however, are ripe for process improvements and additional incremental automation that will allow that number to shrink dramatically. There are two interesting aspects that came out of this however.

First, when you're implementing a business process management solution like this you should probably educate the business that there will be some low hanging fruit that comes out of the statistics generated after the release. As a result, they'll probably want to budget some resources for addressing these things in a final phase of the project. In our case, the project team is set to disband and the resources were not allocated and so the low hanging fruit will have to wait for another project. When that will exactly be is yet to be determined.

It's extrememly important to define before the project both the strategy for collection and the kinds of metrics you want to collect. The obvious reason is that these metrics will embody the questions that the business will ask and therefore point to where the process improvements can take place. In our implementation we did not do a thorough enough job of either collection or understanding and as a result there are some holes in our ability to ask the right questions. We're attempting to address some of this in a followup project but as you can imagine it's more difficult at this stage than it would have been if integrated into the early iterations of the project.

Fun in Omaha

2006-10-26T08:13:00.000-07:00

Having a great time this morning at the Heartland Developer's Conference at the Qwest Center in Omaha Nebraska. Later today I'll be giving a session titled "Real SOA: Implementation and Best Practices" that discusses many of the topics found on this blog along with the experiences of the team over the almost 18 months since I came on board at Compassion as a Solutions Architect.

In short, the talk is broken into seven sections that highlight mostly what went right but also a little of what went wrong or things we could have done better. Those sections and the major point for each follow:

#1 Define a Services Platform
IT buy-in and support is critical

#2 Define Messaging and Schema Standards
It’s not just about coding standards

#3 Consider Cross Cutting Concerns
Build it for the long haul

#4 Consider Logging and Monitoring Up Front
Loosely coupled messaging is hard to tie together

#5 Build a Reference Model
Don’t jump in to business development

#6 Build a Reusable Framework
Budget time to help the rest of the organization

#7 Don’t forget business activity monitoring
Think about questions that will be asked at the end of the day

Always Learning

2006-07-29T09:06:00.000-07:00

Last week we rolled out the last release in this year long project that developed and then proved out the service oriented infrastructure. Although we have issues to deal with related to the proper implementation of some of the uses cases (more on that in a later post) I thought I'd take a few minutes to discuss what I think is perhaps one of our biggest lessons learned.

In designing the contracts for our services we purposely wanted to introduce a layer of abstraction into the contracts so as to avoid creating service interfaces that were too closely tied to the existing enterprise data structure. Of course that's a balancing act since on the one had you need to have enough information represented in the various entity schemas to accurately identify the entity, and on the other hand not include extraneous attributes that serve to clutter the picture.

What we did well in this process is create the larger logical model where we introduced three high level concepts - Constituents, Needs, and Relationships. The first represents individuals of interest to Compassion, the second are entities whose needs can be satisfied such as a child to sponsor, a fund to donate to, or even an activity like praying for the work of the ministry. Relationships then represent the various connections between Constituents and Needs. For example, a Prayer Relationship is established between a Constituent and a Prayer Need while a Payment Relationship is established between a Constituent and a Child, Project, or Fund Need. This structure served us well because it allowed us to subdivide and represent each action for a constituent with a particular relationship type and more importantly allow our messaging infrastructure in BizTalk to recognize these relationship types and create and farm out work to independent orchestrations through a publish and subscribe model. This is an extensible model so that as new relationships are defined the infrastructure can handle them in a loosely coupled fashion (in fact, we implemented a new kind of action in this release without any modifications at all to the core BizTalk messaging model). So far so good.

The problem entered because going in we (and by that I mean I) didn't really understand how the logical model within the Payment Relationship should be defined. In other words, the Payment Relationship being the most complex, it needed a more robust definition at the outset in order to handle all of its variance. Unfortunately we rolled out the first definition of the Payment Relationship schema in our previous release when we were dealing only with relatively simple relationships that did not involve funded donations and commitments (in other words, we were dealing only with promises to pay and not persisting and handling rules for credit card and ACH transactions). And while there were place holders in the schema for this kind of information in the first version, the basic structure didn't extend as well as expected to handle this extended information. As a result, and here is the big idea, we ended up adding attributes to the schema that were too closely tied to the existing relational data model. To put it simply, too many codes right out of the database and not enough abstraction between the messaging model and the persistence layer. What happens when you do this of course is that you not only bring along the codes but their interrelationships and dependencies as well. Had we done more analysis up front, we would have seen these relationships and probably created higher level abstractions to handle them within our service contracts.

In one sense I'm glad we could roll out such a sophisticated system in a series of four iterations over a year, but in another the architectural tradeoffs that resulted from this approach (since we did not have the resources nor the time to fully flush this out ahead of time) will likely hamper us in ways as move down the road. But then again you're always living and learning and so we've toyed with the idea of version 2 service operations in some cases that provide these higher level abstractions.

Transitioning

2006-07-10T13:25:00.000-07:00

For our project we're now about to release our final deliverable. This is the fourth deliverable in the project with the first one being in August of last year. This one is the largest and is used to automate many of the financial transactions performed on our web site which will in turn be used in the automation of requests that come from other channels (scanned forms and external call centers).

That said, the frameworks that the team has built are about to be used by other project teams as well. To that end one of the talks I attended at TechEd was a discussion of on assisting other developers in using framework that you build. One of the points made was that when you’re developing frameworks (like we have for services, orchestrations, and human workflow) the level of effort for framework developers doesn’t necessarily go down over time if you continue to refine it. The illustration used was the following graph.

As a result, the next effort for the team will be to transition from a project team doing business development into a team that continues to refactor and improve the frameworks that other developers use.

One of the other points made, however, was that teams should invest the time in the framework up front in order to make it consumable through the use of code generation, templates, etc. The presenter specifically mentioned that trying to put something together in a just in time fashion when other teams need it doesn’t work well in his experience. Unfortunately, because of the very tight schedule we've been on the past year that is largely where we are. The key insight for other teams doing this kind of work is to budget the time (I know how difficult that is when the work is viewed as "overhead" of course) in each release to refactor your frameworks and (reference architecture) and incorporate guidance as you go as well as allow time for mentoring other teams.

Encryption Handler

2006-06-15T09:50:00.000-07:00

As promised awhile back we'll take a look at the EDAF handler we created in order to encrypt and decrypt data within our messages.

To do so we built the handler in an assembly called Compassion.Services.EDAF.Handlers.Security.dll. This is an AtomicHandler (it doesn't hold state between invocations) that can be configured either in the before or after pipeline of service action calls in order to encrypt or decrypt messages using various key providers.

For example, the configuration of the handler in the ProcessConstituentRequest façade pipeline in order to encrypt messages before they are saved in the ConstituentRequest and ServiceManager databases.


<before>
 <handler handlerName="MessageEncryptor">
  <ci:loggerSettings localValue="US" 
    keyDir="C:\CompassionServices\Config\      
     Keys\CIPublicOnlyKey.xml" 
    operation="Encrypt"  
    message="Request"  
    provider=
      "Compassion.Services.Security.Common.KeyFileProvider,
    Compassion.Services.Security.Common, 
    Version=1.0.0.0, Culture=neutral, 
    PublicKeyToken=7674363010ccf400" />
 </handler>
</before>

As you can see the handler is configured in the before pipeline and is set to encrypt the Request message using the KeyFileProvider. The handler uses reflection to load the key provider dynamically and pass in constructor arguments such as the file path and asymmetric algorithm. At this time the handler hard codes the instantiation of the RSACryptoServiceProvider since it is the only AsymmetricAlgorithm implemented in v1.1 of the .NET Framework.

In order to encrypt data on the way out of a service call (for example for an operation that selects payment data the handler would be configured like so:


<after>
 <handler handlerName="MessageEncryptor">
  <ci:loggerSettings localValue="US" 
   keyDir=
     "C:\CompassionServices\Config\Keys      
      CIPublicOnlyKey.xml" 
   operation="Encrypt"  
   message="Response"  
   provider=
    "Compassion.Services.Security.Common.KeyFileProvider,
   Compassion.Services.Security.Common, 
   Version=1.0.0.0, Culture=neutral, 
   PublicKeyToken=7674363010ccf400" />
 </handler>
</after>

One of the interesting things about the implementation of this handler is that the ObjectEncryptor deals with objects whereas EDAF pipelines pass XmlDocument instances to their handlers via the Execute method. As a result, the handler is responsible for pulling the type information out of the EDAF configuration file and using it deserialize the type and pass it to the encryptor.

In the previous post on this topic a reader asked why we didn't use WS-Security in order to encrypt the message. That's a great question and the answer is that if we had used WS-Security we could certainly have encypted the message body but this would not have allowed us to view the message when it was logged in our infrastructure. In other words, we want visibility into the all parts of the message except those explicitly marked as needing protection.

ESBs and Microsoft

2006-06-14T05:40:00.000-07:00

Day four at TechEd and spent this morning in a session on "Developing the Next Generation ESB on the Microsoft Platform" which was a case study of sorts of a project donw at Kaiser Permanente.

They (three speakers shared the podium) started the talk with a discussion of just what is an Enterprise Service Bus? A 2006 survey found the following characteristics as most important:

Message Broker

Message Transformation Service

Message Validation

Message-Oriented Middleware

The Microsoft speaker then noted that given this list Microsoft will not come out with an "ESB product" because they feel their products are already a superset of the tools required with BizTalk, VS, MSMQ, SQL Server, Host Integration Server, and MOM, along with the guidance they are putting out.

What was interesting about the talk, highlighted by Brian Loesgen, was that for this particular solution that can be used to literally connect thousands of applications, they created a Core ESB Engine that features an envelope (schema) that specified at the meta data level the kind of message entering the ESB as well as what needed to be done with it. From there they dynamically performed both mapping and runtime endpoint resolution (routing). In other words the architecture supported adding new maps and new endpoints without recompiling or redeploying any part of the ESB. In addition they developed a message oriented approach to exception handling whereby developers publish exception messages and handlers (for example SharePoint) subscribe to them. This allows for highly targeted process-specific handlers to be created for custom exception recovery operations.

The final speaker from Kaiser highlighted the client and server integration modules (CIM and SIM) that they use to provide abstraction for clients and servers with regards to policy, location, management, security, etc. The CIM is analgous to our Service Agent Framework while the SIM is used for interoperability as well since they support both SOAP invocation of services as well as the use of BizTalk adapters and other custom mechanisms. He also briefly showed their management console they can use to monitor services and develop SLAs.

Now obviously an organization could build an ESB in this way and the architecture seems more than solid. The problem, and why an organization might be driven to adopt another vendor, is that this solution was developed with the help both of Microsoft and a systems integrator at I'm sure a significant cost in terms of time and money. For smaller organizations or those that can't invest the time required to develop such a solution going to these lengths is not really an option.

Services on CF

2006-06-13T12:40:00.000-07:00

Attended a session on the .NET Compact Framework 2.0 this afternoon. I haven't looked at CF since I came to Compassion over a year ago and so I was quite disappointed to discover that the System.Configuration namespace is not implemented in the new version. For us this is problematic since our Service Agent Framework relies on IConfigurationSectionHandler in order to read information about the service agent types, mock messages, and logging that the framework does on the client.

As a result, if we want to expose our services to mobile clients we'll need to write a Mobile Service Agent Framework that relies on custom XML configuration files as described in the book I wrote with Jon Box back in 2003. We can't simply use a web reference proxy since we also rely on custom SOAP headers that the proxy does not create.

Pragmatic Architecture

2006-06-13T07:43:00.000-07:00

"Pragmatism urges us to look to the consequences of what we do, which the discipline of architecture infused with an idealistic focus on intentions frequently resists"

Attended a pretty entertaining presentation titled "Pragmatic Architecture" by Ted Neward and Cathi Geri. Because the room was heavily populated with architects they had this humorous definition of just what an architect is:

An architect is someone who...

Defines architecture

Gets paid more than real developers do

Focuses on issues that have nothing to do with real-world problems

Thinks in terms of code not clouds

Lost the respect of developer friends & teammates

Speaks with big words and power point slides

Has upper management fooled

But more seriously they went on to talk about the roles or personas that architects take on which include Infraastructure, Enterprise, and Solution (responsible for the design of one or more applications or services within an organization and which the remainder of their talk focused on) and identified by Simon Guest.

They then laid out a taxonomy solution architects can use as they think through laying out solutions...

Communication/Distribution - Patterns of communication such as request/response and the technologies used to move data.

Presentation/Interaction - Style (graphical, console, prgrammatic), implementation, stovepipe vs. composite/mash-up (one front end, many back ends) vs. jewel (many front ends, one backend), and perspectives (user, admin).

State Management - Lifetime, "shape" of the state (relational vs. object vs. hierarchical), location (where, access, and transparency)

Processing - Implementation, concurrent/parallel, transactional

Resource Management - Locator/registry, discovery (plug-and-play, peer-to-peer), injected, a priori knowledge

Evolving to Patterns

2006-06-13T06:21:00.000-07:00

Day three of TechEd 2006 and this morning I attended a session on "Evolving to Patterns" by James Newkirk. What I found most interesting was his use of the "broken windows" paradigm and its application it to software development (others have done so as well). For those not familiar, the idea came from an article written by James Q. Wilson called "Broken Windows" that appeared in the March 1982 issue of Atlantic Monthly. In a nutshell, the idea is that human beings tend not to respect that appear to be in a state of disrepair leading to a vicious cycle of further deterioration. The example used by Wilson is that of an apartment building with one broken window that acts as a signal to vandals to break even more. The same idea can be applied to litter and even the accumulation of junk or dishes in the sink in your own house. The lesson is to fix problems when they are small.

As applied to software development the notion of fixing your broken windows can be distilled into three basic ideas.

Remove Duplication. Duplicated code is broken code because it encourages sloppiness and thought processes where developers don't look for ways to inject abstraction.

Simplify Your Code. Code that is difficult to understand from a structural perspective is broken code because developers would rather write something themselves than have to figure out how to refactor existing code to get the job done.

Clarify your Code's Intent. A corollary to the above is that the intent of the code needs to be clear. Code's whose intent is not clear through the judicious use of comments and naming conventions is broken because it leads to misusing the code or once again the tendency to put that code in a virtual black box that is not to be touched which leads to duplication.

And of course evolving your code to use patterns helps us to design code that is ready to handle the changes that will inevitably come.

Moving to Agile

2006-06-12T10:04:00.000-07:00

Well, day two at TechEd 2006 in Boston and just came out of a talk on implementing agile methodologies in your organization. What I found interesting is that many of the best practices the speakers (from the patterns and practices group) discussed were ones that our project team are currently implementing. As a result, it was a good check for us in seeing what we have yet to tackle both from a team and organizational perspective. Briefly, here are a few of the practices mentioned and what we've done:

Building a team space. While our team members still have cubes, 90% of their work is done in a conference room dedicated to the project. In my view, that has increased the speed of communication and helps us to resolve issues more quickly. It also has the effect of allowing everyone to be more or less on the same page with how things are implemented in areas that they perhaps are not currently working in.

15 minute rule. Essentially this rule says that if you are blocked for 15 minutes, give up and go ask. The team space definitely encourages this behavior although we haven't talked about it specifically. The corollary to this is that when someone asks for help you can't say no. In our organization this doesn't tend to be a problem anyway but surfacing the need to keep the communication open is key.

Real-time code review, aka Pair Programming. We don’t pair program today but we do have some overlapping of assignments with two core BizTalk developers and two services devs. In some cases they work together on a feature and of course I assist and help talk through the implementation of code. We should perhaps be more intentional about this.

Give team time to gel. One of the things that has helped us be successful is that our team had a two month time period in which we implemented a reference architecture before turning our attention to actual production code. As a result, we were able to find our roles before the pressure ramped up. While not intentional, to me, this turned out to be a very helpful by product of the process.

Get a coach or mentor. None of the team had had much exposure to agile methods before and could have used a bit of coaching before the project commenced. That said, we're not really officially doing agile within Compassion and so this probably wasn't an option. But if we had to do it again, a bit more knowledge would have been useful.

Encourage questions and discussion. The team space really makes this possible. It tends to keep the environment open and questions are often raised within the room to no one in particular. To me, that's a sign that people are comfortable with each other and no one is too intimidated to ask.

Stack rank work by importance to customer. While our customers have been very involved in the project, they have only stack ranked work related to hot fixes once various releases have gone into production. Our organization has some way to go it seems before customers are entrusted to do this. The corollary to this is reevaluating that stack frequently. This also needs some work.

Developer estimates are not to be trusted. The speakers were adamant that developer estimates of more than three days are probably essentially guesses. Unless you can break down a feature into very small tasks, then you're not really estimating. Our developer estimates are a bit larger than that and so we probably have some work to do.

Clarify Customer Requirements. We have used use case documents and a feature matrix in order communicate the requirements. Unfortunately, the documents are not always as clear as they could be largely because the conceptual model used in the development of the use cases was not familiar to the developers or the users. Had we to do it again, the use cases would be written in the language of the users and not in the language of the logical model. The team would then be responsible for translating that language into a new model used to implement the software. The speakers also note that automated tools should be evaluated that allow the requirements to be executable.

Customer (or their proxy) has to be available. One of the strengths of our project is that the business users and stake holders almost all reside in the same building and we have had for the most part a business analyst that does a great job of communicating. That allows the team to surface an issue and get feedback very quickly. The average turnaround time is something like four hours.

Implement the 3 strikes rule. In a nutshell this means you don't refactor code until you have three consumers for it. This has been a balancing act for our team since most of what we're building is being done for the express purpose of reuse across future projects. However, the features within the services we build we evaluate and implement based only on the certainty that they will be reused. In other words, our strategy has been to implement the infrastructure that will allow us to refactor when necessary but not to over-engineer in the near term. Examples include the business rules infrastructure we've implemented as well as the exception reporting and suppression techniques we've used. Interestingly, they noted that when you need to refactor, one technique to allow developers to integrate it into the project is to position it as "making room" for a feature before implementation. Their point is that the term "refactoring" implies rework which signals a red flag among some.

Insist on clarity of code. As with most agilists, their emphasis was on code readability for maintenance and refactoring. We insist on NDoc comments and code standards in terms of naming conventions etc. to assist with this but should probably be even more diligent about this.

Reward individuals for team behavior. Our team celebrates success after releases and there is very little in the way of individual rewards. The goal is to get the team to own the entire pie and not just their little piece of it. In that regard, something we do need to work on is the concept of developers working on features and not owning them. We tend to segment our devs into three buckets and each bucket's owns a set of features. While to some extent it is very difficult for everyone to know everything, we should do a better job of specifically asking devs to make changes in code that they have not had a hand in developing. This would increase team ownership.

Everyone is entitled to participate. In our team room we include not only devs but also testers, the project manager, infrastructure support, and technical writing. Including everyone certainly helps with communication issues but the speakers were actually suggesting that these other disciplines take part in real-time code reviews and design sessions. Each person would then contribute at their level. We certainly have not done this explicitly but again, having everyone in the room provides them accessibility to those discussions. At the end of the project we also do lessons learned off site sessions where we collect feedback to help us improve the process going forward.

Compassion International Back on Television

2006-06-01T13:04:00.000-07:00

For those who aren't aware of what Compassion does, there will a TV program airing soon. You can find a list of stations and times here. The following is the press release sent out a few days ago.

From a technical perspective we're interested in seeing how well our service oriented infrastructure handles the additional anticipated load so stay tuned.

COLORADO SPRINGS, CO, May 30, 2006 – Beginning June 1, Compassion International is breaking back into the television scene. Compassion will air a 30-minute television special in major markets throughout the United States. Hosted by contemporary Christian music artist Rebecca St. James, the special will focus on Compassion’s child sponsorship program.

In the 1980s, Compassion was a pioneer among Christian ministries in using television programming, but rising costs eventually led to discontinuing this venue. Today, with hundreds of television channels available, this venue has again become a cost-effective way to share the message of children in poverty.

“This television special creates a wonderful opportunity to share the needs of children in poverty around the world with many people throughout the United States who may never have heard Compassion’s message before,” said Rick Davis, Compassion’s Director of Marketing. “Our hope is that by telling Compassion’s story through a new venue we will help thousands of people to make a difference in the life of a child in need.”

The television special is co-hosted by Tim Glenn, Compassion’s U.S. Advocacy Director, and also features nationally renowned author and speaker Tony Campolo and Hollywood actors Victoria Jackson and Cary Elwes. All participants in the special sponsor children of their own through Compassion’s child sponsorship program.

A complete listing of airtimes can be found on Compassion’s Web site. Visit www.compassion.com and click on the link to the television program.

Compassion International is one of the world’s largest Christian child development organizations, working with more than 65 denominations and thousands of indigenous church partners in Africa, Asia, Central and South America and the Caribbean. Since 1952, Compassion has touched the lives of more than one million children.

It has been recognized for its financial integrity with top ratings and recommendations by several of the nation’s leading not-for-profit “watchdog” organizations. Recently Compassion was recognized by Charity Navigator for its fifth consecutive four-star rating, putting it in the top one percent of 5,000 charities.

For information about sponsoring a child, contact Compassion online at www.compassion.com or by calling (800) 336-7676, Monday through Friday, 8 a.m. to 5 p.m., MST.

1,000,000 Invocations

2006-05-05T13:51:00.000-07:00

On April 25th our service oriented infrastructure recorded it's one millionth service call since our first deliverable rollout in August of 2005. I'd have to say that the combination of Windows Server 2003, IIS, SQL Server 2000, EDAF, and our own EDAF customizations (handlers and modifications to adapters) have provded to be remarkably stable. During that time we've been able to handle almost 40,000 business transactions during that time frame and most importantly automatically handle upwards of 65% of them.

Message Security

2006-05-05T13:28:00.000-07:00

As we gear up for the final deliverable (code-named Bread & Butter) of our project, one of the requirements we've implemented is message security.

Essentially, in this deliverable we'll be dealing with financial information and in order to comply with the Payment Card Industry Data Security Standard we need to be able to protect credit card and bank account information. This is also a part of a larger security initiative within the organization whereby data elements are tagged with a protection level and treated accordingly and includes the deployment of a PKI (public key infrastructure) solution.

Although that initiative is not yet in full swing we've implemented the ablility within our services to tag specific schema elements and attributes as requiring protection and then encrypting and decrypting those elements and attributes using public key cryptography.

Our solution includes the following components:

Encryption components – these are set of common components with a pluggable interface that allows for the encryption and decryption of fields and properties within .NET objects using public key encryption (in our first release just RSA is supported).

Message Encryption Handler – an EDAF handler that can be used to encrypt or decrypt messages sent to and from services

In the remainder of this post we'll look at the encryption components.

Encryption Components
The set of encryption components we've built are implemented in two assemblies named Compassion.Services.Security.Attributes.dll and Compassion.Services.Security.Common.dll. The types that are implemented in the two assemblies are discussed below.

Compassion.Services.Security.Attribute
This assembly contains the following types.

ProtectedDataAttribute class – this attribute can be used to decorate fields or properties within an object like so.


[ProtectedData(EncryptionFormat.Inline)]
public string AccountNumber;

EncryptionFormat enumeration – this enumeration is used to specify which encryption technique will be used to encrypt the data within the field or property marked with the attribute. Valid values include Inline and XMLEnc. Inline has been implemented in this release and will specify that data is encrypted directly within the parent element as in the following example.


<CreditCardPaymentSourceData>
 <CardholderName>Daniel Ducat</CardholderName>
 <ExpirationDate>2007-03-31T00:00:00.0000000-07:00</ExpirationDate>
 <AccountNumber>AkxcG3qHALhDzqd+5Gxrod1tb2q7V8tYQ5MyJxfk/oFd+J0HkPPM
    +J67hMfq2XMgw97kmuDKqnlhiGA7KE3SNkLNAkvmPMwuWUCn7djOEPLKHb+7MDkN
    4JAX2rIGvgr0N1etazudfegNtGRg3fhLp2Nl8lTD8O/KY5gHVA051mU=
 </AccountNumber>
 <CardType>Visa</CardType>
 <Description>****1111</Description>
</CreditCardPaymentSourceData>

The value XMLEnc will be used to specify that the W3C XML Encryption Syntax and Processing specification will be used For example, that specification allows for the encryption of individual elements like so:



<?xml version='1.0'?>
<PaymentInfo xmlns='http://example.org/paymentv2'>
  <Name>John Smith</Name>
  <CreditCard Limit='5,000' Currency='USD'>
    <Number>4019 2445 0277 5567</Number>
    <Issuer>Example Bank</Issuer>
    <Expiration>04/02</Expiration>
  </CreditCard>
</PaymentInfo>

Where the CreditCard element is then encrypted as:


  
<?xml version='1.0'?>
<PaymentInfo xmlns='http://example.org/paymentv2'>
  <Name>John Smith</Name>
  <CreditCard Limit='5,000' Currency='USD'>
    <EncryptedData
     xmlns='http://www.w3.org/2001/04/xmlenc#'
     Type='http://www.w3.org/2001/04/xmlenc#Content'>
      <CipherData>
        <CipherValue>A23B45C56</CipherValue>
      </CipherData>
    </EncryptedData>
  </CreditCard>
</PaymentInfo>

The .NET Framework 2.0 exposes objects in the System.Security.Cryptography.Xml namespace to perform this encryption automatically. In order to implement this in the .NET Framework v1.1 the .NET component that will be implemented must perform the encryption behind the scenes. When we move to .NET 2.0 we’ll likely want to reimplement this class accordingly.

Compassion.Services.Security.Common
This assembly contains the following types.

IAsymmetricKeyProvider interface – this interface provides the methods used to retrieve key information and return a class derived from the System.Security.Cryptography.AsymmetricAlgorithm .NET type. The idea here is to abstract the retrieval of key information from the implementation of the encryption so that different providers may be implemented in the future. For example, the PKI project in our organization would provide a PKI server and key information could be retrieved using a provider that implements this interface.

KeyFileProvider class – because there is no PKI infrastructure in place today the provider that was built knows how to retrieve key information from XML files given the AsymmetricAlgorithm to use and a path. Application will configure the paths to the key information and protect those keys using NTFS permissions.

AsymmetricEncryptor class – this class uses the key provider to encrypt and decrypt data and can do so for both strings and byte arrays. This class may be used independently of the ObjectEncryptor by applications.

ObjectEncryptor class – this class uses the AsymmetricEncryptor to encrypt and decrypt objects which have fields or properties marked with the ProtectedDataAttribute. Clients such as the MessageEncryptionHandler we'll discuss in the next post may use this object to encrypt and decrypt messages as in the following example:


RSACryptoServiceProvider aag =
new RSACryptoServiceProvider();
KeyFileProvider kfPublic = new KeyFileProvider(aag,
@"C:\PublicOnlyKey.xml");
ObjectEncryptor.Encrypt(pcr, kfPublic);

Here the client instantiates the asymmetric algorithm to use, passes that into the IAsymmetricKeyProvider (in this case along with the path to the key) and finally uses the Encrypt method of the ObjectEncryptor to encrypt the object. Since the object is passed by reference the object will contain the newly encrypted data.

Next time we'll look at the handler we implemented so that service operations can encrypt and decrypt data.

Services and .NET 2.0

2006-04-07T09:41:00.000-07:00

At long last I've started to look into porting our various frameworks to version 2.0 of the .NET Framework.

As a first step I was able to recompile the entire Enterprise Development Application Framework (EDAF) which included our customizations to the web service interface adapter and transaction handler. While I encounter numerous warnings (mostly related to changes in the System.Configuration namespace and how the assembly key file is specified in the project) there were no compilation errors. Next, I compiled our custom handlers (logging, rules, monitoring). All of that worked well so I was able to deploy EDAF to a new server hosting ASP.NET 2.0.

From there I was able to re-generate all the .NET code from our schemas and then reubild our common classes (such as the DataFactory) and base classes and saw virtually no warning and no errors. Finally, I built the Constituent Management Service that we use to handle sponsor information and deployed it to the server which once again went smoothly.

From my machine I then used our existing v1.1 Service Agent for the Constituent Service to make a call to the SelectConstituent operation. Moving forward this is the way our main clients including BizTalk orchestrations and the Ultimus BPM Studio will be consuming the services.

It worked!

From here it's just a matter of checking functionality for the various operations but it looks very promising thus far.

Exception Handling and Suppression

2006-04-07T07:38:00.000-07:00

One of the interesting architectural issues we had to grapple with when developing our service oriented infrastructure was how to handle exceptions from our services.

Obviously one simple approach would simply have been to throw .NET exceptions from our service code in EDAF and then allow ASP.NET to translate that into a SOAP fault. However, this approach does not allow us to differentiate between business (the child you chose to sponsor is no longer available) and system (the database was down) exceptions, does not allow us to add additional information to the exception, does not allow for the return of relevant information in the main body of the response along with the exception, and does not allow overriding of certain exceptions.

To handle these issues we designed an Exception schema that looks as follows.


<xs:schema
xmlns="http://schemas.compassion.com/common/
exceptions/2005-03-01">
 <xs:element name="Exceptions">
   <xs:complexType>
     <xs:sequence>
       <xs:element minOccurs="1" maxOccurs="unbounded" 
             name="Exception">
         <xs:complexType>
           <xs:sequence>
             <xs:element minOccurs="1" maxOccurs="1" 
             name="DateTimestamp" type="xs:dateTime"/>
             <xs:element minOccurs="0" maxOccurs="1" 
             name="Source" type="xs:string" />
             <xs:element minOccurs="0" maxOccurs="1" 
             name="Code" type="xs:string" />
             <xs:element minOccurs="1" maxOccurs="1" 
             name="Message" type="xs:string" />
             <xs:element minOccurs="0" maxOccurs="1" 
             name="StackTrace" type="xs:string" />
           </xs:sequence>
           <xs:attribute name="Type" type="xs:string" 
             use="required" />
           <xs:attribute name="Level" type="xs:int" 
             use="required" />
         </xs:complexType>
       </xs:element>
       <xs:any minOccurs="0" maxOccurs="unbounded" 
         namespace="##local" processContents="lax" />
     </xs:sequence>
     <xs:attribute name="SchemaVersion" 
       type="xs:decimal" 
       use="optional" />
     <xs:anyAttribute namespace="#any" 
       processContents="lax" />
   </xs:complexType>
 </xs:element>
</xs:schema>

As you can see this schema looks similar to what you'd find in a .NET exception inherited from System.Exception with the addition of Type and Level as well as the ability to add additional data with the inclusion of the any element and attribute per our schema standards.

You'll also note that this schema includes the notion of multiple exceptions through the root Exceptions element that includes one or more Exception elements.

This schemas is then added to the response envelope schemas for each service operations like so:


<xs:element minOccurs="0" maxOccurs="1" 
 ref="ex:Exceptions" />

In this way service actions can populate these exception arrays as they see fit, for example in response to a business violation.


ExceptionsException exception = 
 new csc.ExceptionsException();
exception.Type="Business";
exception.Level=0;
exception.Source = System.Reflection.Assembly.
   GetExecutingAssembly().Location;
exception.StackTrace = Environment.StackTrace;
exception.Message = string.Format(
 this.ResourceManager.GetString("MultipleAddressesFound"),
 correctedAddresses.Length);
exception.Code=this.ResourceManager.GetString(
   "MultipleAddressesFoundCode");
exception.DateTimestamp = DateTime.Now;
this.Exceptions.Add(exception);

The typical service then uses a resource file to retrieve the actual error message based on the culture found in the SOAP header. We also have a base class method that can build one of these exceptions when given a .NET exception.

You'll notice that the base class also allows for the collection of the exceptions to be tracked using an array list so that the service can simply add the exception to the collection and continue processing.

The precense of exceptions is what then triggers whether or not the service attempts to rollback any transactions that have started and what the response code will be. This the typical code block we use near the end of our services.


if (this.Exceptions.Count > 0)
{
 //append exceptions
 response.Exceptions = this.GetExceptions();  
 response.ResponseCode = 
  ResponseCode.Failure.ToString();
 //invalidate trx.
 this.InvalidateTransaction();
}

Here the exceptions that have been collected get placed into the response object to be serialized back to the caller, the response code is set to Failure (which allows clients to detemrine whether or not they should inspect the exceptions collection since we're not throwing a SOAP fault), and the base class method is called to mark the transaction is being invalid (behind the scenes this method uses an EDAF context property that is read by the transaction handler configured in the pipeline for the business action).

The second interesting aspect of how this works is the Level attribute. We've added this attribute so that we can tag our exceptions being thrown in the business actions as follows:

Business Critical and System = 0

Business Overridable = 10

Business Warning = 20

Business Informational = 30

The idea here is that if a consumer of a service receives a failure response and inspects the exceptions, they have the option of submitting the request again but this time setting an ExceptionSuppressionLevel configured on the envelope of the request message like so:


<xs:attribute name="ExceptionSuppressionLevel" 
  type="xs:int" use="required" />

This attribute will default to 0 as will the level in any exceptions that are created in business actions.

The consumer will then populate the ExceptionSuppressionLevel on the call and the service will suppress any exceptions at that level or above.

For example, in the case where one of our UI's wishes to create a constituent and override the non-critical exceptions the call would set the ExceptionSuppressionLevel to 10 and the call might return with a Success response and avoid throwing the exception.

The interesting aspect of this architecture is that it allows us in the future to have services that return a mix of information including informational messages. Exception Levels 0-20 will implicitly roll back the work of the business action while level 30 never does. Using an integer rather than an enumeration allows us to be flexible as other levels of exceptions arise.

Sponsoring Children and the SOA

2006-03-15T10:10:00.000-08:00

Last Friday morning the third release (the first two were at the end of August and middle of December) of our SOA-related project, code named "Dill" went into production.

For me, the most exciting aspect of the release is that we are beginning to automate the linking of sponsors with children they wish to sponsor by selecting the child on the web. Our system runs a variety of verification rules on both the child and the sponsor (surfaced through operations of a "Need Management Service" where a child is logically defined as a "Need"). If the service operations flag issues that need to be resolved (for example, the child's curernt photo is not available or their biography hasn't been updated) we then send the request to Ultimus where a member of Compassion's Commitment Team can resolve it.

Thus far the system is running smoothly and the team has not seen any systems exceptions (which we define as exceptions thrown by services that are unhandled and therefore end up in a System Exception queue in Ultimus and can be retried once the issue has been corrected).

Today, the team is meeting off-site to discuss lessons learned from this release and review the use case documentation for later releases.

The two biggest issues we had with this release were that...

1) the release encompassed two different business areas which meant the development of BizTalk orchestrations and service operations for both tracks simultaneously. As a result we had two BizTalk developers and two service developers working on their separate tracks. However, we have one Ultimus subject matter expert (SME) and one database development SME who were stretched pretty thin. Not to mention me on the architecture side of the house trying to juggle the technical and business requirements for both sides (which I didn't always do very well). As a result, during the user acceptance testing we logged 52 issues: 21 were coding errors, 1 was a configuration error, 10 were related to missing business requirements, and 20 were missing development requirements (meaning that the business process was understood but all the details were not communicated to developers, mostly my responsibility)

2) we changed a few of the core components, for example our controller orchestration that manages the long running business transaction and fixed issues with correlation which meant that either we deploy all of our components side by side or re-process the requests. We chose the latter since this afforded us the opportunity not to have to retry the system exceptions that were already in the system, clean out all the running orchestrations in BizTalk and incidents in Ultimus, and take advantage of new functionality related to schema changes.

I wrote a little tool that extracted the core documents from our production environment. When we deployed we had 836 current requests in the system and so we used the tool to export these documents (in production we have tracking turned on for this document type).

The core of the code for this tool uses WMI to retrieve the documents and save them to a directory. The UI is very simple and asks for the name of an orchestration for which to extract messages and a path where the documents are saved.


int status = 0;
string statusSQL = string.Empty;
string ServiceName = txtOrchestration.Text;
int i = 0;

switch (cbStatus.SelectedItem.ToString())
{
  case "All":
    break;
  case "Active":
    status = 2;
    break;
  case "Ready To Run":
    status = 1;
    break;
  case "Suspended Resumable":
    status = 4;
    break;
  case "Dehydrated":
    status = 8;
    break;
  case "Completed With Discarded Messages":
    status = 16;
    break;
  case "Suspended Non-Resumable":
    status = 32;
    break;
}

if (status != 0)
{
  statusSQL = " and ServiceStatus = "
   + status.ToString();
}

ManagementObjectSearcher services =
  new ManagementObjectSearcher(
  "\\root\\MicrosoftBizTalkServer",
  "select * from MSBTS_ServiceInstance where ServiceName='"
  + ServiceName + "'" + statusSQL);

int docs = Convert.ToInt32(txtMax.Text);
bool done = false;
   
foreach(ManagementObject o in services.Get())
{
  ManagementObjectSearcher mos =
   new ManagementObjectSearcher(
   "root\\MicrosoftBizTalkServer",
   "select * from MSBTS_MessageInstance
    where ServiceInstanceID='"
   + o.GetPropertyValue("InstanceID").ToString()
   + "'");

   foreach(ManagementObject mo in mos.Get())
   {
     i++;
     mo.InvokeMethod("SaveToFile",
      new object[]{txtPath.Text});
     sbPanel1.Text = "Extracting message "
      + i.ToString();
     if (i == docs)
     {
       done = true;
       break;
     }
   }

   if (done ==true)
 break;
}

sbPanel1.Text = "Done. Extracted " + i.ToString()
  + " messages.";

When the document is extracted BizTalk saves both a .OUT file which contains the XML document and an .XML file which contains the context properties which include our message and message stream ids used in our custom SOAP headers.

We also then have a tool which uses our Service Agent Framework to resubmit these documents. The core of the


private void RunIt(FileInfo f)
{
  try
  {
    label1.Text = "Processing " + f.FullName;
    this.Refresh();

    XmlDocument payload = new XmlDocument();
    payload.Load(f.FullName);

    XmlNodeList nodes = payload.GetElementsByTagName(
     "ConstituentProcessRequest",
     "http://schemas.compassion.com/constituent/
       processrequest/2005-10-01");
    XmlDocument xdoc = new XmlDocument();
    xdoc.LoadXml(nodes[0].OuterXml );

    // add the updateable attribute to the profiles
    AddUpdateable(xdoc);
    AddPRType(xdoc);

    XmlDocument resp = ServiceAgentProxy.ExecuteOperation(
     "Compassion.Services.Agent.
     Constituent.ConstituentServiceAgent",
     "ProcessConstituentRequest",xdoc,Guid.NewGuid().ToString(),
     Guid.NewGuid().ToString(),Guid.NewGuid().ToString(),
     "performance","urn:CPR Load Utility",String.Empty );
  }
  catch (Exception ex)
  {
    listBox1.Items.Add(f.FullName + " " + ex.Message );
    System.Windows.Forms.MessageBox.Show(ex.Message);
  }
}

The call to the ExecuteOperation method of the ServiceAgentProxy object submits the document to the service facade (the ProcessConstituentRequest operation of the Constituent Management Service) then then uses MSMQT to submit the document to BizTalk.

The other interesting thing that this tool does is to modify the documents where appropriate in order to support some of the minor schema changes we made. As a result we wrote couple helper methods that add new attributes to the document where needed.

We'll then use this tool to submit the 836 requests to the infrastructure over the next few days.

Dill is Near

2006-02-23T12:01:00.000-08:00

Just a quick update on our project.

We're currently in the process of performing integration testing in preparation for our next release in early March code-named "Dill". That release will feature the deployment of three new services that contain 13 new service operations implemented as EDAF business actions. Of those one service and two operations are Infrastructure related and provide a service facade for Compassion's internal Consituent Communications Queue (CCQ) system while the other two services and 11 operations are Enttiy Services. Behind those services are two new business processes implemented in BizTalk and three new Ultimus exceptions.

One of the interesting things I was looking at the other day was that since our first release went into production last fall our services have handled over 625,000 requests, 25,000 of which are business transactions submitted from our web site. Of those 77% are following the happy path and updating our enterprise database automatically.

Previously 100% of the requests were handled manually and so the combination of web serivces for entity manipulation, BizTalk for business process automation, and Ultimus for human workflow have combined to form an effective tool for automating a portion of Compasison's business which in turn frees our internal employees to perform more strategic activities, and ultimately helps to release children from poverty. To me anyway, that's an appropriate use of technology.

BizTalk Pipelines in the SOA

2006-02-23T11:35:00.000-08:00

Here's an interesting problem that occurred on our team that we weren't aware of.

In our architecture all messages that in and out of BizTalk using custom SOAP headers based on WS-Addressing as I discussed in Message Standards.

What that means is that we implemented a custom pipeline component in C# that parsed our SOAP header and promoted the various fields we use to track messages such as MessageId and MessageStreamId. This pipeline is invoked when messages are received from outside sources such as our human workflow tool, Ultimus BPM Studio which posts to an HTTP receive location.

In the Exeucte method in our original code we used the following line to get the load the incoming document


xmlDoc.Load(inmsg.BodyPart.Data);

This code works fine as long as tracking is turned on in BizTalk. Once we turned off tracking in development we ran into problems where the properties were not being promoted like so:


There was a failure executing the receive pipeline: 
"Compassion.BizTalk.Pipelines.Common.
SOAPHeaderReceivePipeline"
Source: "Compassion.BizTalk.Pipelines.Utilities" 
Receive Location: "/UltimusResponse/BTSHTTPReceive.dll" 
Reason: Pipeline component exception - Not implemented

Through a call to Microsoft (and subsequently through reading things like this) we discovered that we should be using the GetOriginalDataStream method in order to grab the message. This is the case since BodyPart.Data clones the incoming stream and the HTTP adapter uses the network stream which does not support cloning. Turning on tracking allows BizTalk to create a wrapper around the original message. And hence you can use BodyPart.Data.

So our new code looks as follows:


Stream s = inmsg.BodyPart.GetOriginalDataStream();
xmlDoc.Load(s);

Once we have the incoming message we can run a series of try/catch blocks as follows to grab the elements we need from the SOAP header and promote them.


// strip off the envelope and return 
//just what is within the Body
System.IO.MemoryStream ms = new System.IO.MemoryStream
  (Encoding.UTF8.GetBytes(xmlDoc.SelectSingleNode(
   "//*[local-name()='Body']").InnerXml));
inmsg.BodyPart.Data = ms;
    
try 
{
  strMessageId = xmlDoc.SelectSingleNode(
   "//*[local-name()='Header']
   /*[local-name()='MessageID']")
   .InnerText;
  // remove the "uddi:" if present
  strMessageId = strMessageId.Replace("uuid:","");
  inmsg.Context.Promote(
   "MessageId", 
   "http://schemas.microsoft.com/
   BizTalk/2003/SOAPHeader".ToString(),
   strMessageId);
}
catch {}

VSTS Whys and Hows Part II

2006-02-14T16:15:00.000-08:00

In the first article in this series I described some of the problems inherent in modern software development and suggested a few ways in which those issues might be addressed by tools like Visual Studio 2005 Team System (VSTS) soon to be released. In other words, the first article addressed the “whys” of VSTS. In this article I’ll tread further on that path by providing an overview of Visual Studio 2005 Team System (VSTS) from a functional perspective so that you can see how VSTS plans to address the problems.

More than the Sum of Its Parts
From a high level, VSTS is composed of five products:

Visual Studio Team Foundation. This is a server product or platform that exposes features shared by the various clients (as well as Office applications such as Microsoft Excel and Microsoft Project) shown in the diagram. While some of the features exposed by the clients can be used independently, it is Team Foundation that pulls them all together and enables the collaboration and communication so desperately needed by software development teams.

Visual Studio Team Foundation Client. A client tool geared towards project managers and other non-developers. The Team Foundation Client exposes a subset of the shared features in Team Foundation through the concept of the Team Explorer discussed below. While the Team Foundation Client is based on the Visual Studio IDE it does not include the language features and other tools that developers and architects would use. The Team Foundation Client is the basis for integration in the other products mentioned here.

Visual Studio Team Architect. The client tool used by software architects to design serviced-oriented solutions and validate their designs against the actual hardware and software environment. Team Architect also includes a modeling tool called the Class Designer that is tightly integrated with code.

Visual Studio Team Developer. The traditional Visual Studio IDE augmented with a set of features that enable developers to more easily comply with policies and collaborate in a team environment. Although many of the features found in Team Developer such as unit testing and static code analysis can be used without access to Team Foundation in the Visual Studio IDE, the metrics or results that the tools output cannot then be shared with the team.

Visual Studio Team Test. A client tool for those devoted to testing that enables them to design, manage, and execute manual, unit and load tests.
Each of these products is designed to provide specific features that aid in the process of software development. It is their combined power when adopted by the entire team, however, that allows VSTS to address the core issues teams face of communication, predictability, integration, and process. In the remainder of this article I’ll drill down on each one of the products and walk through its key features so you’ll understand how it fits into the bigger picture.

Visual Studio Team Foundation
As we saw in the first article, many of the problems encountered by software development teams arise from lack of communication within the team and lack of integration of the tools used by the team. In VSTS the Team Foundation provides the server platform that enables both communication and integration. In that way Team Foundation can be thought of as the organizing principal in VSTS. And because of its centrality it is worth having a look at its major features.

Change Management. A brand new (not simply a new version of Visual SourceSafe) source code control infrastructure based on SQL Server 2005 for versioning code and other project deliverables. Built from the ground up for secure and high-speed access, this is a robust version control system that supports advanced features such as branching, change sets, a novel feature called shelving, as well as check-in policies. To give one example of how this feature addresses the problem of integration discussed in the previous article, consider VSTS’s check-in policies. Essentially, when the Project Manager creates the project in VSTS he can associate policies with the check-in of source code. These policies validate that a developer’s changes comply with whatever the organizational requirements are before the set of changes can be checked-in. For example, a check-in policy might be activated so that a developer must run static code analysis or a unit test on his code before checking it in. If the developer fails to do this he’ll receive a policy warning in the Team Developer IDE. If he chooses to ignore the policy the project administrator can optionally be notified through email.

Work Item Tracking. A centralized repository stored in Team Foundation’s SQL Server 2005 operational data store and used to track tasks, defects, requirements, enhancements and - through the use of custom attributes - anything else required by the project. If Team Foundation is the organizing principal in VSTS, then Work Item Tracking is the central concept within Team Foundation. Requirements, unit tests, test cases, and source code can all be associated with work items to provide the basis for a rich set of metrics collected by the client tools. And as with other aspects of Team Foundation, a set of default Work Item types ships with the product but can be extended by your organization or third parties. Project managers and other team members can interact with work items through Visual Studio Team Foundation Client or even Microsoft Excel and Microsoft Project. Of course, correlating this information with the detailed data collected by the various tools makes for a powerful combination in terms of reporting on the health, status, and direction of the project.

Reporting. A set of over thirty predefined reports built with SQL Reporting Services used to report on the health and status of the project. These reports can then be made available both on the Team Project Portal and directly within the applications used by the team members through the Team Explorer. Data to produce these metrics is actually collected behind the scenes by the tool in which the team members work as they perform their tasks and then stored in a central SQL Server Analysis Server repository from which the reports are created. For example, data about test execution and results are saved when a tester executes his test cases and data concerning code churn is saved when developers check out and check in code files. This approach has the obvious advantage of freeing Project Managers and other team members from the hassle of data collection and ensures that the data is always fresh. At the same time the data collection process is not intrusive and works within the natural workflow of the team.

Team Project Portal. To help solve the problems related to collaboration and communication Team Foundation automatically creates a web site or Team Project Portal for each project using a Windows SharePoint Services (WSS) team site. From this portal project stakeholders can get a quick view of the project’s progress both freeing up the time project managers would spend creating status reports by hand and enabling stakeholders to provide input and make course corrections. The portal’s document library is also pre-populated with document templates, for example discovery and requirements specification templates, and sample files for use by team members.

Project Management. We saw in the first article that teams could benefit from tools that help guide them through the process of software development. In order to provide process guidance VSTS includes the concept of methodology templates which can be thought of as a schema for software project. These templates help to structure the process used by the team by exposing a set of documents, project roles, work item types, a web portal template, policies, and what are called process tasks from within all of the VSTS tools. For teams that have no process a Project Manager can choose one of the predefined methodology templates that will ship with VSTS such as a new version of Microsoft’s Solution Framework called MSFT Agile.

Build. One of the required activities in a software development project is to actually build the software. To that end a core feature of VSTS exposes a common process for building executables based on the tool Microsoft uses internally called MSBuild. This system is designed to deliver a “build lab in a box” that makes it easier for teams to implement the best practice of daily builds. This tool allows for the creation and execution of automatic builds on a separate build server that integrate different types of tests. As the tests execute data is collected about the builds in the operational and analytical data stores, for example to tie the build number to Work Items and automatically create work items for build failures.
Taken as a whole these features provide a solid platform on which teams can define and implement their software development process.

Team Foundation Client
The second product that comprises VSTS is the Team Foundation Client. This can be thought of as a tool that Project Managers and other generally non-technical team members will use. While it runs within the Visual Studio shell, it does not include the overhead of language compilers and modeling features that PMs and other don’t need nor want.

The services of Team Foundation are exposed in the Team Foundation Client through the Team Explorer. This tool is accessible within all of the different VSTS clients and really provides the foundational access to VSTS.

The Team Explorer is a tool window that can be hidden similar to the Server Explorer that developers are already familiar with in previous versions of Visual Studio .NET. Through this window team members can easily find work products and data associated with the project

More importantly the Team Explorer also acts as a launch point into the set of tools that ship with VSTS. By simply navigating to the appropriate node in the Explorer the context of the application changes and allows the team member to create and run reports, query on tasks assigned to them, create custom queries, and view project builds and test results among other activities. No longer is there a need to manually switch between various tools thereby increasing productivity.

Team Architect
In the previous article I discussed the scenario where a Solution Architect creates a design only to find out later that it is precluded by the hardware or software configuration of the deployment environment. In order to address these sorts of issues the Team Architect product, in addition to including the Visio UML modeling tools present in the prior version of Visual Studio Enterprise Architect, includes a suite of graphical modeling tools. These tools are designed to allow architects to visually design systems based on service oriented architectures (SOA) and to validate those designs against the actual environments in which they’ll run.
Specifically, the suite of tools includes:

Application Designer. The Application Designer provides a design surface on which the architect can diagram a set of applications that exchange messages. In other words the Application Designer can be used to create a model of connected applications that are dropped on the design surface and connected via Web Services Description Language (WSDL) contracts. Off the shelf application types include web sites, Windows applications, web services, databases, and BizTalk services as you can see in the toolbox on the left side of the screen.

Logical Data Center Designer. The Logical Data Center Designer is a designer that an Infrastructure Architect can use to create a logical model of a part of the data center. As such it allows for the creation of “logical servers” that specify the application hosting environment on those servers. For example, a logical server can specify the communications protocols allowed (perhaps on HTTPS) and the types of services that are enabled (IIS, FTP) in addition to the communication between other logical servers and those servers grouped into what are called “zones”. Constraints including security requirements on the logical server can then be specified. The architect can either create the servers himself or read the configuration information from physical machines using the tool.

System Designer. The System Designer provides a design surface used by the architect to create what is referred to as a configured system. A configured system is composed of one or more applications defined in the Application Designer for a particular deployment. The System Designer provides a higher-level view so that architects can visualize how systems will communicate with external systems or with other internal (nested) systems. From there the architect can drill down to the applications if necessary in order to define them.

Deployment Designer. The Deployment Designer is used by architects to create a deployment for a configured system defined in the System Designer or for an application defined in the Application Designer. Here is where the application and systems models are mapped onto the data center models through a process called binding. Prior to this all the information (metadata) collected by the designers was made available to a constraint engine and as a result validation and consistency checks can be made. If the configuration of an application violates constraints on a logical server, the Task List within Team Architect alerts the architect to the issue, thereby allowing the problem to be corrected before actually attempting a deployment. The Deployment Designer also generates a deployment report visible that can be used to communicate between IT and Operations as well as a deployment script that includes configuration settings and files to be deployed.

Class Designer. Although not directly connected to the other three tools, the Class Designer provides the architect with a superset of the capabilities of traditional UML static structure or class diagrams. The Class Designer represents the .NET type system with full fidelity and therefore allows architects to be very precise in the design of their classes, structures, enumerations, and interfaces in a way that UML cannot. Indeed, as types are designed on the surface of the Class Designer the code for those types is written in the project. Conversely, changes to the code will be reflected in the model since both the code and model are simply reflections of the SDM. And because design at this level isn’t restricted to architects, the Class Designer is also available in the Team Developer. Obviously, this more connected approach can help to solve some of the communication problems inherent in software development.

Team Developer
In many teams today, however, developers are hindered from reaching high levels of productivity by having to cobble together multiple tools in order to adopt development best practices and having to repeatedly context-switch between tools in order to interact with requirements and tasks. When multiplied across all the developers on the team, this in turn drives down the predictability of the entire project. To address problems like these Team Developer includes the Team Explorer mentioned previously. This tool window immediately gives the developer the ability to see which Work Items (tasks) they are responsible for and through its integration with Team Foundation can tie actual work the developer is doing to those work items thereby automatically and continuously updating the project’s status. As you would expect Team Developer integrates with Team Foundation by providing the interface to Team Foundation’s change management (source code control) system and its’ check-in policies discussed previously.

Team Developer also includes the following tools.

Unit Testing and Code Coverage Analysis. One of the best practices that Team Developer enables is unit testing and by extension the practice adopted by many agile teams known as Test Driven Development or TDD. Team Developer does this through its inclusion of an attribute-based unit testing framework much like the popular NUnit that is directly accessible within the IDE. Using this framework developers can create unit tests and execute them during the development process while behind the scenes the results are automatically recorded in Team Foundation’s analytical data store. In addition Team Developer enhances its power by coupling it with the testing best practice of code coverage analysis. Using this practice, when a team member executes a unit test not only will the test results be tabulated but optionally the percentage of code executed during the test for each source code module (files, namespaces, classes, methods) can be viewed as well as being stored by Team Foundation for later use.

Static and Dynamic Code Analyzers. A second best practice that teams often have a harder time getting their developers to adopt is code reviews. When done well, code reviews with other developers and architects provide a structured way to gain feedback in a constructive way that proactively enables the developers to make changes that improve code quality and maintainability. To address this need Team Developer includes both Static and Dynamic Code Analyzers that can be thought of as an automated method of performing code reviews. For example, the static code analysis tool comes preconfigured with a set of warnings that can identify known issues in .NET development involving performance, security, general design, and naming guidelines akin to the FxCop tool in use by many in the .NET developer community.

Code Profiler. Developers can also increase their productivity through proactively addressing performance problems using tools Team Developer includes. With these tools developers can sample the performance and then instrument their code in order to collect detailed performance statistics to enable drill down on potential problem areas. For example, a developer can use their unit tests to generate a Performance Session where the code is sampled in order to identify potential performance bottlenecks in terms of both throughput and resource usage. From there the developer can instruct the tool to insert timing probes into the desired subset of code to gather more detailed statistics. Finally, a detailed performance report is generated so that developers can decide where to begin further investigation.

Team Test
VSTS takes testing a step further than simple unit testing by making them available in the Team Test product geared specifically for testers. This product enables testers to execute the unit tests developers create in a systematic fashion to ensure that code checked in by multiple developers passes previously constructed tests. This is an example of another testing best practice referred to as regression testing.

Team Test goes even further by making testers first-class citizens of the project by highlighting the importance of testing at the team level through a test authoring and execution environment. That environment includes additional test types such as Web Tests, Load Tests (similar to the tool Application Center Test tool that ships with Visual Studio .NET 2003), Manual Tests, and Ordered Tests and test case management all integrated into Team Foundation.

VSTS Whys and Hows

2006-02-02T11:55:00.000-08:00

As I’m sure many readers are aware by now, Visual Studio developers are about a month away from the second part of the Visual Studio 2005 release that includes a slew of new features that make a developer’s job easier. Visual Studio 2005 Team System.

In the first of this two part series I’ll describe a few of the specific problems you and your team probably face that served as the impetus for the creation of Visual Studio 2005 Team System (VSTS). In the next article I’ll get down to how VSTS is designed to address these issues. Taken together, the goal of these two articles is not simply to give you a laundry list of VSTS features, but rather to place them in the context of the problems they were designed to address.

Software Development is Hard
I don’t want to sound like a grumpy old man who sits in front of the barbershop longing for yesterday while muttering about failures of the modern world, but software development is hard and getting harder.

Gone are the simpler days of monolithic applications running on a single platform free from deployment issues. Gone are the heady days of client/server computing and the “enterprise data model” accessible through Visual Basic or Powerbuilder apps free from data sharing issues. And even gone are the days of simple data-driven ASP or ASP.NET web applications accessing our data behind our firewall. No, today in our ever more connected world, our solutions require access from multiple platforms, devices (heterogeneity), and networks, data sharing across the enterprise and with business partners (service orientation), and scalability on commodity hardware. All of this in an environment where the driving factors are less time and fewer resources.

The truth of this trend towards increasing complexity has driven many organizations to consider adopting the more loosely-coupled architectural pattern of service orientation, or a Service Oriented Architecture (SOA), made practical by the implementation and maturation of web services. And as with any architectural shift SOA and web service applications come a new set of design and implementation challenges that developers will need to address.

I know that this come as no surprise to many of you reading this article - you who have managed, designed, built, and tested complex enterprise solutions requiring heterogeneity, service orientation, and scalability. But I also know that in doing so many of you are looking for tools to make your jobs easier

So why is software development difficult? In short I see four primary reasons teams struggle to build quality software.

Communication Breakdowns
The reality is that in many cases IT teams are today both geographically and functionally distributed. This distribution creates gaps in communication that provide opportunities for issues to be dropped, misunderstandings to arise, and information flow to be slow and haphazard.

Fortunately as you’ll see in the next article, with VSTS the very technology that makes geographical distribution possible in many cases can also be applied to open up new channels of communication. And of course the benefits of tools enabling better communication aren’t restricted to geographically disparate teams. Even in small and local teams, crucial information is often dispersed and not easy to find or not captured in the first place. Tools that provide a centralized location for all of this information is in many ways the first step towards controlling the project.

At the same time, as with much of the modern world, the IT industry has tended towards increasing specialization. Although in a small organization you may have to be a jack of all trades, in many organizations silos have developed where expertise in project management is restricted to one group while development resources come from another and solution architectures from a third. Because some of these groups share certain attributes and not others there are differing amounts of conceptual space between these groups as illustrated in Figure 1 where Developers and Testers are more closely related than Solution Architects and Project Managers. This specialization over time has the effect of producing conflicting best practices, architectures, and ultimately a vision for how software should be developed.

The Proximity of IT Silos

As an example of addressing the first need consider the case where a Solution Architect is designing a system all alone, safely ensconced in his architectural silo made of ivory. When his architecture is complete he hands it off to the developers who merrily code away. Job done. Not so fast. When it comes time to discuss how this solution will actually be deployed in production, the Solution Architect is shocked and dismayed to discover that Operations doesn’t support the server or communications configuration he was assuming for his “perfectly architected” solution. And of course Operations derives a certain guilty pleasure in crushing the ivory tower. His solution no longer looks so perfect and he then must rework the architecture or spend political capitol in trying to get the Operations policies changed, costing both time and development resources.

What our architect needed in this scenario was a tool that helped him map his solution architecture to the existing hardware and software environment including the Operations policy so that he could take it into consideration during his work. In the second article in this series you’ll see how VSTS addresses this problem through a suite of design tools.

Tools that enable information flow between these groups will go a long ways towards solving communication breakdowns.

Lack of Predictability
The Danish physicist Niels Bohr once said that “prediction is very difficult, especially about the future”. I don’t think he had software development in mind but I do think it is applicable. This is borne out intuitively by the concern many have in the IT industry for being able to predict the success of projects. Their concern is not unfounded. As Steve McConnell writes in his 2004 book Professional Software Development:

“Roughly 25 percent of all projects fail outright, and the typical project is 100 percent over budget at the point it’s cancelled. Fifty percent of projects are delivered late, over budget, or with less functionality than desired.”

Why do these projects end up over budget, late, feature-poor, or cancelled? In most cases it was because their teams couldn’t accurately predict or control the software development lifecycle. And at the heart of enabling better prediction in your projects lies metrics and repeatable practices.

The simple fact is that teams manage projects according to the metrics they are able to collect. If the metrics collected are the right ones the Project Manager and other stakeholders can quickly get a feel for where the project is headed, to better estimate its completion, and to actually drive it towards completion by analyzing risk and making course corrections when necessary. In addition, a key point that organizations often undervalue is that good (meaning real and not anecdotal) metrics on previous projects rather than personal memory, subjective intuition, or seat of the pants guesswork provide the best input into making estimates on future projects.

I would suggest that what project teams could use are tools that help them address the area of metrics in at least the following ways:

By suggesting predefined metrics taken from best practices that have been proven to provide insight into projects such as metrics related to progress towards the schedule, the stability of the plan, code quality, and measuring the effectiveness of testing.

By offering a single place in which data to produce metrics is stored in order to eliminate the separate silos in which that data is stored today.

By supporting an automatic way of collecting data in a timely fashion that is integrated into the natural workflow of the team.

The second aspect of more accurately predicting the success of software projects lies in the adoption of repeatable or “best” practices. While we all know that one size does not fit all I also know that in almost any endeavor applying a structured or methodical approach reduces the variability in the outcomes.

Unfortunately, many teams have no set of repeatable or structured practices that they can hang their hats on and they can use to focus their efforts. For these teams, for example, a developer is simply tasked with implementing a feature and then left to his own devices. While some developers may naturally or through intentional analysis apply some set of structured principles to their development, many will not and the result is that the productivity of those developers and hence the predictability of both the software quality and the schedule are almost impossible to get a handle on. This is largely borne out by the oft-repeated finding that developers differ in their productivity on the order of 10 to 1 as noted by McConnell in Professional Software Development. In other words, some developers are 10 times more productive than others. And just as the adoption of uniform approaches to training and conditioning has shrunk the variation in times between competitors in athletic competitions such as track and field, I think at least part of the difference in the productivity of software developers can be overcome by applying repeatable and proven practices to the software development process.

Tools can help in this regard by integrating structured practices into the normal workflow of developers, for example, by including a unit testing framework.

Lack of Integration
The primary cause of the problem is that over time many organizations have accumulated a collection of software development lifecycle (SDLC) tools through purchase, acquisition or merger, or even custom development. Unfortunately, in most cases these tools weren’t designed to be used together and are not integrated with the IDEs in use in the organization. As a result, this lack of integration can manifest itself in myriad ways but often bubbles to the surface as follows:

The data collected by the tools they use in their SDLC exist in separate silos and is not easily integrated or related to data from other tools.

The tools they use don’t provide a platform and device independent way to implement extensibility and custom tools.

Using “best of breed” tools forces their team members to context-switch between tools in order to perform functions of their process.

The tools they use don’t know about the policies or constraints of their process. In other words, their tools cannot interact in any meaningful way with the processes they do have.

Obviously, the size of the overall integration problem is more often than not proportional to the size of the organization you’re in. However, even small organizations are not immune from integration issues as even a single relied upon tool that isn’t integrated can manifest all of the above issues.

One approach to this problem, and that taken by VSTS as you’ll see in the next article, would be to integrate the lifecycle tools directly into the software that team members use in their day to day work. For example, rather than require a developer to go to a web site to too see which features he’s been assigned why not make those features visible from inside his IDE? And further, why not have the tool associate the feature with the actual source code and unit tests he’s working on so that metrics can be automatically collected on the implementation of the task.
A second way in which lack of integration rears its ugly head is when tools don’t work together to surface or expose policies adopted by the organization. Typically these policies or guidelines that attempt to regulate the software development process exist only in three-ring binders or if you’re fortunate on an Intranet. Unfortunately, in both cases out of sight is out of mind and so the policies end up being enforced across the organization only sporadically if at all.

What is needed is for tools to enforce policies dynamically during the regular work flow of team members. One of the bets Microsoft is making with VSTS is that once development tools automate process guidance, then most of the overhead associated with the process as well as the majority of the resistance to compliance will evaporate.

It is this kind of deep integration, both in terms of your team member’s workflow and tools, can improve the productivity of you teams and the quality of the software you produce.

Lack of Support for the Process
Trying to coordinate the people, geographies, roles, and tools involved in projects while at the same time tracking and making predictions about the interaction of these aspects would be a challenge for anyone. So how do teams solve these sorts of problems and raise their level of success?

The standard answer to that question has been to adopt a software development methodology such as the Rational Unified Process (RUP) or Extreme Programming (XP) or a customized version of one of these or other standard approaches. While this approach can be and often is effective, most developers don’t have access to tools that surface or provide visibility into the process so that they can more easily follow it. And because of the lack of tools support, the perceived complexity of some of these methodologies and simple inertia, some teams simply haven’t adopted any methodology or process at all. Unfortunately, these teams must instead fly by the seat of their pants much of the time. What is needed are tools that provide the overarching structure for projects in a way that teams can flexibly integrate their own process or use a predefined process that guides them on their way.

Because many teams (primarily those in smaller organizations who don’t have the resources or expertise to implement a rigorous process) have not adopted a software development process there is ample opportunity for tools to provide guidance and therefore improve the quality and timeliness of software produced by these teams.

Of course, the majority of larger organizations have already adopted a methodology and so their challenge is in making their process visible through the tools they use.

What is needed is for tools to allow for the customization of boiler plate processes or the building of custom processes that integrate with the tools.

Summing it Up
In this article I’ve discussed a number of common problems teams face when developing solutions in the increasingly complex and constrained world of software development and hinted at how VSTS will address them.

What is needed in order to solve these kinds of problems what is needed is not just another upgrade of their integrated development environment, but instead an “integrated services environment” that encompasses the entire extended software development team.

At its core this integrated services environment, in order to be successful, needs to accomplish three things:

Reduce the complexity of delivering modern service-oriented solutions

Be tightly integrated and facilitate better team collaboration and communication

Enable customization and extensibility by organizations and ISVs

Exactly how VSTS is constructed to make this happen will have to wait for the next article.

Using Local Transactions

2006-01-31T14:28:00.000-08:00

In my last post I talked a bit about how the DataFactory class we're using is structured. In the past I've written about how we implement transactional behavior across business actions by utilizing the EDAF Context and Enterprise Services through the EDAF transaction handler.

Of course within a business action we also need to implement transactional behavior, for example, when an action makes calls to multiple stored procedures that need to be implemented in a single unit of work. To do this we could use simply configure the transaction handler in the pipeline for the business action. However, doing so incurs additional overhead because of the precense of the handler and the need to start a new MSDTC transaction.

To implement a lighter weight approach we've added support for local transactions to our DataFactory. The DataFactory exposes a BeginTransaction method that opens a database connection and return an object that implements the IDbTransaction interface. The methods of the factory are then overloaded to accept the transaction object and therefore can use it across multiple calls.

So the pattern for using local transactions is....

// get the data factory to use
// (a property of our BusinessObjectBase class)
DataFactory df = this.GetDataFactory(FactoryType.Write);

IDbTransaction tx = null;
IDbConnection connection=null;
int ret = 0;

try
{
  // create a transaction to use
  tx = df.BeginTransaction(IsolationLevel.ReadCommitted);
  // grab the connection since the Rollback wipes out
  // the property of the transaction object
  connection = tx.Connection;
              
  HybridDictionary parms = new HybridDictionary();
  parms.Add("partner_country_code",
    req.CCQOrder.PartnerCountryCode);
  // etc.

  df.ExecuteNonQuery("InsertCCQNeed",parms,out ret,tx);

  // call a second sproc
  HybridDictionary ccqParms = new HybridDictionary();
  ccqParms.Add("partner_country_code",
    req.CCQOrder.PartnerCountryCode);

  df.ExecuteNonQuery("InsertCCQ",ccqParms,out ret,tx);

  // commit the transaction
  tx.Commit();

}
catch (System.Exception ex)
{
  if (tx != null)
  {
    if (tx.Connection.State == ConnectionState.Open)
      tx.Rollback();
  }
}
finally
{
  if (connection != null)
  {
    // release to the pool
    if (connection.State == ConnectionState.Open)
      connection.Close();
  }
}

Data Access Helpers

2006-01-26T12:39:00.000-08:00

Today I thought I'd drill into one of the core components we use in our service oriented infrastructure and invoked from inside our EDAF business actions and business components that implement our service operations. Like most applications, in order communicate with our enterprise database we use a data access helper component written in .NET v1.1 called the DataFactory. First though, a short digression on the nature of data access helpers.

Architecturally, Data Access Helpers are Data Access Logic (DAL) Components that are internal to the Data Services Layer. As such, these components are called from the bodies of methods in the components that make up the public interface of the Data Services Layer. As a result, they are used via composition with other Data Access Logic components and primarily for the following reasons:

Code Simplification. By abstracting code that is often duplicated in similar classes, developers can be more productive by having less code to write.
Provider Abstraction. Since ADO.NET is built on the concept of .NET Data Providers, or specific concrete classes used to connect to either a narrow (like SQL Server or Oracle) or broad (like ODBC and OLE DB) data source, data access helpers can abstract the provider code thereby allowing the code to execute against different providers without recompilation.
Database Interface. Data Access helpers can also incorporate code to dynamically discover and call database object such as stored procedures. In this way, data access helpers provide an interface to the database.

In addition to the MSDN Data Access Application Blocks, you can use a variety of home grown approaches, two of which I've discussed more fully in chapter 18 of Teach Yourself ADO.NET in 21 Days. In order to understand how our DataFactory works let's first review it's poor cousin the ProviderFactory.

Provider Factory
This class of approximately 120 lines is a variant of the Abstract Factory Pattern documented by the GoF. This class abstracts the creation of .NET Data Provider specific objects into a single class that exposes factory methods for each of the four primary kinds of concrete classes in the provider.

The parent object then simply needs to create an instance of the ProviderFactory and call the factory’s methods in place of the instantiation of concrete classes. For example:

IDbCommand com = _pf.CreateCommand(“usp_GetProduct”, con);

As a result, there is basically a one to one correspondence between the code with or without the ProviderFactory.

Internally, the ProviderFactory relies on the Activator.CreateInstance method to create objects of the appropriate type and cast them to the generic interface supported in the System.Data namespace.

For example, the code for the CreateCommand method is as follows:


Public Function CreateCommand(ByVal cmdText As String,
  _ ByVal connection As IDbConnection) As IDbCommand
  Dim c As IDbCommand
  c = CType(Activator.CreateInstance(_comType(_pType), _
    False), IDbCommand)
  c.CommandText = cmdText
  c.Connection = connection
  Return c
End Function

You’ll notice that the CreateInstance method accepts the type of object to create stored in the array _comType. The array is defined as a private class level variable like so:

Private _comType(3) As Type

And populated in the instance constructor as follows:


_comType(ProviderType.SqlClient) = GetType(SqlCommand)
_comType(ProviderType.OLEDB) = GetType(OleDbCommand)
_comType(ProviderType.ODBC) = GetType(OdbcCommand)
_comType(ProviderType.Oracle) = GetType(OracleCommand)

The ProviderType enumeration is a custom enumeration that references the supported providers. In this case the four providers that ship with VS .NET 2003 are hardcoded into the constructor.

Now let's move on to the DataFactory and discuss the three key features of the factory including:

Provider Abstraction. Since the data factory needn’t expose provider specific objects for the most part, the technique used to abstract the provider is different from that used in the provider factory. Although it could also be rolled into the provider factory class. The data factory class is hard-coded to use any of the four providers that ship with VS .NET 2003 but can extended through a configuration file to include other providers.
Database Independence. Independence from syntax particular to a database server is accomplished in the data factory through the use of XML statement files. These files include the database specific syntax (i.e. the actual Transact-SQL statements used against SQL Server) to define the commands that get executed. Statement files can be used both with stored procedures and inline SQL.
Caching. Internally, the data factory caches instantiated command objects in a shared Hashtable that is checked each time a command is executed. In order to alleviate contention for the command object, its ICloneable interface is used to create copies of the object when it is needed.

The data factory class exposes the following members.

*CacheFilePath
*Connection
*Provider
*UseCache
BeginTransaction
CreateDataAdapter
CreateSqlFiles (shared)
CreateSqlFile (shared)
ExecuteDataReader
ExecuteNonQuery
ExecuteScalar
ExecuteSqlXmlReader
GetDataSet
GetDataTable
GetProviders (shared)
GetXsltResults
RemoveStatement
SyncDataSet
SqlFilesCreated (event)

The “Execute” methods execute statements and either don’t return results or return a reader that must be traversed. The “Get” methods return complete objects.

The class makes heavy use of overloading to supply varying signatures for many of the methods. For example, the ExecuteNonQuery method includes four overloaded signatures defined as follows:

public int ExecuteNonQuery(string statement, HybridDictionary parms)

public int ExecuteNonQuery(string statement, HybridDictionary parms,
ref int returnVal)

public int ExecuteNonQuery(string statement, HybridDictionary parms,
ref int returnVal, IDbTransaction transaction)

public int ExecuteNonQuery(string statement, HybridDictionary parms,
 ref int returnVal, IDbTransaction transaction,
 ref Object[] outputParms)

Note that the statement name is passed into the methods as a string while the parameters to the statement are passed in a HybridDictionary object.

Note: A System.Collections.Specialized.HybridDictionary object uses a ListDictionary for internal storage when the number of items in the collection is small and a Hashtable when the number of items is large. This is the case since the Hashtable is not as efficient for storing a small number of elements.

The Provider property, which can be passed to the constructor or is defaulted to SqlClient, is used by the private _createProviderTypes method to create Type objects to hold the provider specific types. It does this either by using a hardcoded case statement or reading from a DataFactory.config file.

Once the types are created it uses the CreateInstance method of the System.Activator class to instantiate the provider specific objects at the appropriate times as shown in the snippet from _createProviderTypes:


Private Sub _createProviderTypes()
 Select Case Me.Provider
  Case "SqlClient"
    _conType = GetType(SqlConnection)
    _comType = GetType(SqlCommand)
    _drType = GetType(SqlDataReader)
    _daType = GetType(SqlDataAdapter)
    _parmType = GetType(SqlParameter)
  Case Else
    ' Load the types from the configuration file
 End Select

 ' Create an instance of the connection object
 _connection = CType(Activator.CreateInstance(_conType, _
   False), IDbConnection)
 _connection.ConnectionString = _connect
End Sub

In order to abstract the vendor specific SQL syntax from the data factory, each SQL command is encapsulated in an XML statement file like that here.


<?xml version="1.0" encoding="utf-8" ?>
<DataFactory>
  <Statement name="GetTitles" type="StoredProcedure">
    <Sql>usp_GetTitles</Sql>
    <Parameters>
     <Param name="author" SqlName="@author" type="string“
       maxLength="30" direction="Input" />
     <Param name="title" SqlName="@title" type="string“
       maxLength="100" direction="Input" />
    </Parameters>
  </Statement>
</DataFactory>

Note that the statement file includes a name attribute that contains the friendly name while the Sql element contains the SQL statement. Each parameter to the statement is defined in a Param element and optionally includes the length, direction, and the SourceColumn to map the parameter to in a DataTable.

In this case a stored procedure is referenced but statement files can also be used to handle inline SQL. For example, the Sql element could be replaced with:

SELECT * FROM Titles WHERE author = @author AND title LIKE @title + ‘%’

And the resulting statement file would be functionally equivalent to that shown on the slide.

Internally, the data factory class uses shared, synchronized, provider-specific Hashtable objects (since there may be multiple instances of the data factory using different providers) to store statement objects that have been parsed into structures using an XmlTextReader. The structure definitions are (in VB):


Friend Structure Statement
  Public Name As String
  Public SQL As String
  Public CommandType As CommandType
  Public Parms As ArrayList
  Public Command As IDbCommand
End Structure

Friend Structure Parm
  Public Name As String
  Public SQLName As String
  Public Type As String
  Public Direction As ParameterDirection
  Public maxLength As Integer
  Public SourceColumn As String
End Structure

The Hashtable object that holds each of the provider-specific Hashtable objects is declared as (in VB):


Private Shared _provCache As Hashtable =
  Hashtable.Synchronized(New Hashtable())

After the statement files have been created and are read into an instance of the Statement structure they are placed in provider-specific Hashtable that is assigned to the instance when the Provider property is set.

When a statement is executed the private _getStatement method does the following:

1. Determine if the statement is in the cache
2. If not it loads and parses the statement file creating the command object with parameters before placing the command in the cache
3. If so then it pulls the command object out of the cache and clones it
4. It then populates the parameters
5. And finally returns the command

The most interesting code in this method is that used to find the statement if it exists and then to instantiate the command object accordingly.


' See if its in the cache
If Not Me.UseCache OrElse _procs.ContainsKey(statement) Then
  ' Pull it out of the cache
  s = CType(_procs.Item(statement), statement)
  newCom = CType(_cloneObject(s.Command), IDbCommand)
Else
  s = _getStatementFromFile(statement)
  ' Build the command, add the parameters
  com = CType(Activator.CreateInstance(_comType, False), _
   IDbCommand)
  com.CommandText = s.SQL
  com.CommandType = s.CommandType
  ' Add the parameters
  ' Add the statement to the cache and clone it
End If
' Populate the parameters
Return newCom

The _cloneObject method relies on the ICloneable interface to create a copy of the command object whos parameters can be populated.

Note: Only providers who command objects support deep copies using the ICloneable interface will work with the data factory.

We've had good success with the DataFactory since it simplifies the code we need to write in our business actions, provides a level of abstraction when stored procedures change, and improves performance by caching command objects.

Localizing Services

2005-12-20T14:37:00.000-08:00

Since our service-oriented infrastructure is meant to be extended for use by partner countries we architected it so that different service consumer's culture settings could be taken into account.

As discussed in my post on Messaging Standards this meant incorporating into our standard SOAP header the Culture and UICulture elements defined as follows:

Culture (Optional) The culture setting for the client specifying the formats of date time, currency, and numeric data that follows RFC 1766 derived from ISO 639-1 and ISO 3166, e.g. US English="en-US"

UICulture (Optional) The culture setting for the client specifying the language to use that follows RFC 1766 derived from ISO 639-1 and ISO 3166, e.g. US English="en-US"

In order to set the culture and to return messages in the appropriate language we then had to effect several areas of our code.

ServiceAgentBase
From the service consumer's perspective a call to a service operation can be made through a Service Agent using our Service Agent Framework. The layer supertype in that framework is our ServiceAgentBase class from which all service agents are defined.

When a method on one of the derived service agent classes is called, the implementation delegates much of the work to a protected method of the ServiceAgentBase such as ExecuteOperation. This method is then responsible for building the SOAP request, choosing a transport, and then sending the request and receiving the response.

In order to build the SOAP message the method uses a set of schema classes we generated using XSD.exe and then modified. For example, we have a class simply called Compassion.Schemas.Common.SoapHeader that encapsulates our header schema. In that class we have the following:


public class SoapHeader
{
  private string _culture = 
   Thread.CurrentThread.CurrentCulture.Name;
  private string _uiculture = 
   Thread.CurrentThread.CurrentUICulture.Name;
...
}

As a result, clients who instantiate service agents can first set the culture on the current thread (typically the culture is already set of course) before doing so and thereby transmit the culture through the SOAP envelope.


Thread.CurrentThread.CurrentCulture = 
 new CultureInfo("de-DE");

Thread.CurrentThread.CurrentUICulture = 
 new CultureInfo("de-DE");

// Create the update request
ConstituentUpdateRequest req = CreateRequest();

// Create the service agent
ConstituentServiceAgent agent = new ConstituentServiceAgent
 ("Compassion.Services.Agent.Constituent.Tests.Unit");

// Send the request and get the response
ConstituentUpdateResponse resp = 
 agent.UpdateConstituent(req);

In this way, by using the service agent consumers will automatically get consideration for their culture when service operations are processed.

WebService Transport
When the SOAP message reaches the server EDAF intercepts the request and processes through one of its transports. Thus far we've used the WebService transport exclusively and so we modified the standard code to be aware of our SOAP headers.

Within the WebServiceInterfaceAdapter.cs file we added code to process our SOAP headers if the headers match our namespace. If so the code unpacks the header and then builds a CIHeader object which it then places in the EDAF Context object like so:


// Create the header if it doesn't exist yet
if (h == null) { h = new csc.CIHeader();}

switch (sun.Element.LocalName) 
{
 case "Culture":
 {
   h.Culture  = sun.Element.InnerText;
   break;
 }
 case "UICulture":
 {
   h.UICulture  = sun.Element.InnerText;
   break;
 }
...
}
//Add the header to the context
_context.Add("ciHeader",h,false);

One of the interesting aspects of this is that if the WebServiceInterfaceAdapter does not find the culture settings in the SOAP header it will try and read them from the underlying transport itself, in this case HTTP like so:


if (h.UICulture == null)
{
  // See if you can find the culture in the HTTP header
  if (HttpContext.Current.Request.UserLanguages !=null) 
  {
    if 
    (HttpContext.Current.Request.UserLanguages.Length > 0)
    {
      // Take the first one in the collection
      h.UICulture =
       HttpContext.Current.Request.UserLanguages
        [0].ToString();
      if (h.Culture == null)
      {
        h.Culture =
        HttpContext.Current.Request.UserLanguages
         [0].ToString();
      }
    }
  }
}

At this point the culture is now captured on the service provider side.

BusinessActionBase
After the request passes through the adapter and any handlers that have been configured it makes it's way to the business action where the actual work is done. All of our business actions are derived from our BusinessActionBase class which as you might expect, contains the code to read the culture information and use the .NET ResourceManager to allow the derived class to pick up the correct strings.

The supertype does this by simply examining the CIHeader object in the Context if it exists and then pulling the values out.


//try to pull it from the ciHeader.
if (Header != null)
{
  if ((Header.Culture != null) && 
   (Header.Culture.Trim().Length>0))
    culture = Header.Culture;
  if ((Header.UICulture != null) && 
   (Header.UICulture.Trim().Length>0))
    uiCulture = Header.UICulture;
}

//try to pull it from the app settings.
if (culture==string.Empty)
  culture = ConfigurationSettings.AppSettings["Culture"];
if (uiCulture==string.Empty)
  uiCulture = ConfigurationSettings.AppSettings
   ["UICulture"];

//call SetCulture
SetCulture("Strings",
 this.GetType().Assembly,strCulture,strUICulture);

You'll also notice that this code includes a last ditch effort to pull the culture settings from the application's configuration file. This allows the implementer of the service to set a default culture other than the neutral culture.

The last line of code above calls the SetCulture method in order to actually set the culture. The key part of the SetCulture method sets the current thread's culture to the codes passed in and instantiates the ResourceManager that is then exposed via a protected property.


Thread.CurrentThread.CurrentCulture = new 
  CultureInfo(strCultureCode);

Thread.CurrentThread.CurrentUICulture = new
  CultureInfo(strUICultureCode);

m_objResourceManager = new ResourceManager
 (strResourceFileBaseName,objCaller);

The end result is that dervied business actions use the exposed resource manager to return exception and other messages to the client in their own language. For example, when an exception is raised the business action would code like the following in order to return an exception code and message to the caller.


exception.Message = 
 this.ResourceManager.GetString("MissingAddress");
exception.Code=
 this.ResourceManager.GetString("MissingAddressCode");

Business Rules in the SOI

2005-12-20T07:39:00.000-08:00

One of the architectural topics that caused our team more than a bit of thinking was our approach to implementing rules.

The original idea for handling rules as noted in the high-level architecture diagram below (to the left of the Business Action) was to implement a common rules engine that would be responsible for the storage and execution of rules. This approach would allow for one-stop shopping for Process and Entity Services as well as a common repository for easily changing rules.

Initially three different options were discussed:

BizTalk Rules Engine (BRE). Since BizTalk is a major player in the Compassion SOA and includes a rules engine, our first thought was to leverage this engine to store and execute rules.

While the engine is versatile and can be called from inside .NET assemblies (such as the business actions we’ll be developing), the major impediment to using the engine was twofold: 1) The BRE does not include an easy to use interface for managing the rules. As a result, other organizations have created their own interfaces using tools like Microsoft Excel to manage the rules. Without doing something like this rules could not be managed by business user, 2) when using the BRE a BizTalk installation is required with the appropriate licensing. This model may not be desirable if services are deployed in a partner country since it would force the adoption of BizTalk at those locations.

From a previous concall with Microsoft it is also not anticipated that Microsoft will ship the BRE as a separate product in the near future.

NxBRE. A second approach that was considered was adopting the open-source rules engine for the .NET Platform called NxBRE, which is a port of the JxBRE engine available at SourceForge. This engine use rules defined in RuleML and allows for rules to be defined using Visio stencils. One of our consultants performed an evaluation of NxBRE to determine how it could be invoked from within business actions.

After evaluating this option it was decided that although it overcomes the licensing restrictions implicit in the use of the BizTalk BRE it still does not provide a simple way for business users to easily change the rules. Further, as with the BizTalk BRE, NxBRE is an inference engine that works off the concepts of Facts, Queries, and Implications. This structure would work well for simple data validation but would be unwieldy when trying to represent data modification rules since a large number of facts would have to be loaded into the engine and the return data from the engine would be complex (for example to represent rules regarding how an entity’s attributes are set based on all of the data associated with the entity).

Ultimus 7.1. Version 7.1 of the BPM suite includes a rules engine called Ultimus Director that can be used from within Ultmus workflows to represent rules. It is designed with end users in mind and so can be used by business users to change and update rules without developer intervention. However, the engine is not meant to be invoked from outside Ultimus and it is assumed that licensing restrictions would apply. As a result, this option does not appear to be a general solution.

In order to clarify Compassion’s position regarding the implementation of business rules in the SOA architecture a conference call with Gartner analyst Jim Sinur was held. He provided the following clarifications and best practices.

1. The need for and use of BREs should be determined based on business rule volatility

That tipping point is usually in the range of 25-35% of the rules to be processed
Compassion should rank business rules by; change frequently, hardly ever change, and middle-of-the-road, that is, perform a volatility analysis
BREs may be cost effective for managing the rules that change frequently
Compassion may have few business rules that change frequently and a BRE would be overkill

2. BREs are expensive and there are currently no solid candidates in a .NET architecture

Virgin Atlantic has created a custom spreadsheet interface to BizTalk for volatile business rules, this allows business analysts to change these rules without developer assistance
Microsoft may add a similar interface to BizTalk, however, this is just something Compassion should watch and not count on

3. Create guidelines for business rule placement

"Flow-Related" business rules should be placed in BizTalk or Ultimus
Business rules should be parameterized whenever possible - especially the more volatile rules
Avoid placing business rules within custom business framework services - especially any "flow-related" rules
Create and use a mechanism for business rule management
Report on what's there
Keep pulse of how often the rules are changed
Change the strategy if you discover a high degree of change
Create rules so they can be tailored to meet global requirements

Approach
As a result of this analysis and the Gartner discussion the decision was made not to pursue adopting a BRE as a single repository for all rules within our service-oriented infratructure. The decision was based primarily on two factors. First, although a volatility analysis has not been done it is highly unlikely that Compassion meets the 25-35% threshold where implementing a BRE would be cost effective. Secondly, within our technology scope there doesn’t appear to be a BRE that could be employed without creating a custom UI and/or incurring licensing costs. As a result, the direction we’re following is to employ the native rules engines within BizTalk and Ultimus (Process Services) and then implement custom rule development within Entity and Infrastructure Services.

In following this approach the following rules matrix was developed in order to provide guidance as to how and where rules would be implemented.

Process Rules
What?
Rules related to the flow of a system to system process or system to human process, for example, a rule that changes the routing of a form within Ultimus based on partner country

Where?
Implemented in the BizTalk BRE or Ultimus Director when available

Data Integrity Rules
What?
Rules related to how data is ultimately stored in a data store such as Compass or an XML document including FK relationships and link tables, for example, a rule that dictates that when an email is updated the shared email addresses are also updated

Where?
Typically implemented in stored procedures that persist the data and invoked from business actions. If the data is persisted in XML documents, then XSDs can be used to implement these rules

Data Modification Rules
What?
Rules related to how data can be inserted, updated, or deleted. Typically, these rules are dependant on the state of the persisted data in a data store, for example, a rules that specifies that in the US only two email addresses are stored

Where?
Implemented directly in business actions that perform inserts, updates, or deletes.

Data Validation RulesWhat?
Rules related to the format of data, for example, the requirement that an email address conform to a regular expression

Where?
Implemented directly in business actions that perform validation typically prior to inserts or updates

In following with the best practices outlined by Gartner the Data Modification and Data Validation rules include support for externalization. Specifically, this support includes:

The ability for a business action to be notified as to which rules to process based on a configurable value (such as the partner country) and on which client is invoking the service

The ability for the business action to use specific parameters for a rule based on a configurable value (such as the partner country)

We implemented this support using a custom EDAF handler as shown in the following diagram.

As indicated in the diagram the rules handler is invoked as a part of the service implementation pipeline for specific business actions. The handler passes configuration information indicating the local value along with the service action name to the rules store. The rules store returns two data structures, one that is a collection of rule identifiers (GUIDs) that should be processed for the business action, and a second that includes a collection of country specific values to use when processing the rules. Both data structures are placed into the EDAF Context. The Rules Handler caches the data structures once retrieved for performance reasons.

The business action then uses the data structures in the Context to determine which rules to process and what values to use when processing. The semantics of the rules themselves are encoded in C# or VB .NET code, typically in private helper methods that make them easy to maintain. The collection of rule ids are used only to indicate which rules to process and so the presence of the id in the context triggers the execution of the rule.

Because the two collections are not related, multiple rules can use the same local value in its processing.

When rules are retrieved the handler queries against these two tables. Given the LocalValue, ServiceActionName and the client making the request (From information encapsulated in the SOAP header), a set of Active RuleIDs would be returned. If the the client doesn’t appear in the BusinessActionClientRules table, the Active flag from the BusinessActionRules table applies. Notice that the syntax of the rule is not included in the table, only the id and whether it is active for the business action and client. This data would then be placed in a hashtable within the EDAF Context object by the Rules Handler.

Inside a business action, particularly the BusinessActionBase class, we've written code to check the hashtable for the RuleID before executing the rule.

We've also built an ASP.NET user interface to manage the rule data.

Overall, this solution has workd quite nicely and in production we use these rules both to provide specific values to use when evaluating a rule and to turn rules on and off for specific clients.

Using NLB

2005-12-15T14:43:00.000-08:00

One of the design requirements for our service-oriented infrastructure was to support a scale out architecture. For our needs we chose to use Microsoft's Network Load Balancing (NLB) system.

Early on we created a reference architecture in order to test the integration between our various components including EDAF services, BizTalk 2004, legacy ASP code, and Ultimus BPM 7.1. In our test environment we set up two clusters, one for the EDAF services and one for the Ultimus services.

In the reference architecture simple tests of both the EDAF services cluster and the Ultimus cluster were performed. In the case of the former 200 requests were sent through one of our orchestrations which ended up invoking our constituent service via the service agent framework we devleoped. In the case of the latter 50 requests that created Ultimus incidents through our facade service at a time were spawned by a test harness. In both cases the tests (using performance monitor) revealed that all requests from a particular client were being serviced by one machine in the cluster. The machines servicing those requests were the servers that had the higher priority set in the NLB configuration in the Host Parameters tab.

NLB settings were then reconfigured from single to "no affinity" and balancing load at 50% in the port rules from within the NLB UI. Tests were then re-executed with no difference in the results.

Details regarding the Microsoft NLB algorithm were consulted. The important paragraph of this document is:

"When inspecting an arriving packet, all hosts simultaneously perform a mapping to quickly determine which host should handle the packet. The mapping uses a randomization function that calculates a host priority based on their IP address, port, and other information. The corresponding host forwards the packet up the network stack to TCP/IP, and the other cluster hosts discard it. The mapping remains unchanged unless the membership of cluster hosts changes, ensuring that a given clients IP address and port will always map to the same cluster host. However, the particular cluster host to which the clients IP address and port map cannot be predetermined since the randomization function takes into account the current and past clusters membership to minimize remappings."

In other words, when “no affinity” is set, the host that services an incoming request is determined by the IP address and client port number in a deterministic fashion based on a randomization algorithm and the number of servers in the cluster.

As a result we looked into how client port numbers are generated on client machines.

It turned out that within our Service Agent framework we have a ServiceAgentBase class from which all servicce agents are derived. Within this class requests are executed using the SoapHttpWebClient class. Internal to this class the following code is in the constructor.


public SoapHttpWebClient(string requestUri)
{
_requestUri = requestUri;
_httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(requestUri);
_httpWebRequest.KeepAlive = false;
_httpWebRequest.Method = "POST";
_httpWebRequest.ContentType = "text/xml;charset=\"utf-8\"";
_httpWebRequest.Accept = "text/xml";
}

Tests using a throw away application and a network monitor utility revealed that requests were load balanced when the System.Net.Sockets.TcpClient was used to generate the requests but not when HttpWebRequest was used as in the constructor code above. This was the case since using TcpClient generates a new connection and therefore a unique port number for each request.

However, by adding the following line of code to the constructor above…


_httpWebRequest.KeepAlive = false;

the HttpWebRequest class creates a new connection for each request which allows the TCP stack on the machine to assign a unique port number to the request. Making this change to the SoapHttpWebClient entails a slight performance penalty on the client machine since connections cannot be reused. As a result, we added a configuration setting, <KeepAlives> to the configuration section for the service agent so that clients have the option of using this setting (with the default set to false).

Inter Action Communication

2005-12-15T14:17:00.000-08:00

One of the interesting issues we tackled in this release was that of allowing EDAF business actions to communicate their state information.

Our original design made the assumption that we would have controller type business actions that would invoke other business actions using the in-process dispatching adapter in EDAF. Each of the invoked business actions would when necessary create exceptions and return those to the controller who would then add them to its exception collection that eventually is sent back to the caller as a series of XML elements.

While this design allows the business actions to be independent and the controller to rollback the work of all the invoked actions, it does not allow for business actions that have dependencies to communicate when called by the controller. In other words the communication model here is essentially from the controller to each invoked action whereas pairs of invoked actions have no visibility to each other.

Fortunately, EDAF allow for this by supporting the concept of a Context object. So the mechanism to do this is to add items to the EDAF Context object (this.Context) before invoking the action. We were using this technique already to send flags like a ValidateOnly flag that instructs each action to validate the request but do not process it.

However, for this release we added the capability for actions to be aware of the exceptions that have been created within the entire request. We use a base class called BusinessActionBase from which all of our actions are derived. To this class we added a protected method called DoesExceptionExist that accepts one of our custom exception codes and looks in the context item with a specific key. If the item doesn’t exist it returns false.


if (this.DoesExceptionExist("COUNTRY_CODE_NOT_FOUND")==false)
{ …

When invoking an action that has dependencies developers can therefore add an ArrayList populated with exceptions (also provided by the base class) to the context before submitting the request.

We added this capability because one of our actions needed to determine whether the COUNTRY_CODE_NOT_FOUND exception was previously thrown. So when action A invokes action B it does the following:


// Added to context so exceptions can be searched
daContext.Add("Exceptions",this.Exceptions);
da.Submit(daContext);

EDAF and our base class handle the rest.

Sweet!

2005-12-15T12:53:00.000-08:00

Today we deployed the second release of our SOA project code-named "Sweet". We have four deliverables remaining and each is named after a type of pickle and include "Dill", "Half-Sour", "Kosher", and "Blue." It's a long story that came out of the requirements gathering process and too much caffeine at an offsite meeting.

This release introduced a new EDAF based service built on the .NET Framework v1.1 to handle payment information and an augmentation of our service that manages constituents. We also introduced a new Ultimus 7.1 workflow to handle reconciling constituents and one updated and four new BizTalk 2004 orchestrations. Overall we deployed code to five application (two web servers, BizTalk, Ultimus, and our EDAF services machines) and four SQL Server database servers.

From a schedule standpoint the work was done primarily by three in house developers and three consultants over a span of a little over two months and with the assistance of a QA manager. In that time the team modified or wrote 49 assemblies, 26 stored procedures, four ASP.NET and one ASP site, and one Ultimus workflow - some very nice work contributed across the board.

Probably the biggest deployment challenge in this release involved introducing new BizTalk orchestrations while needing to keep existing production instances running. We had one core orchestration in our first release which was processing each request (which take on average 48 hours to complete with human interaction in Ultimus). In the Sweet release that orchestration became merely a component of a larger process as we incorporated the processing of new transaction types. So we had to create a new version of that orchestration that returned a response document among other things.

So when we deployed we had over 100 orchestrations we suspended before deploying the new assemblies. Once we deployed we had to add a binding policy in the GAC to one of our schema assemblies that had changed and then we were able to resume the existing orchestrations (as an aside I had originally thought that deploying a new schema assembly would be problematic based on other things I had read but it turned out to be quite painless). As users in Ultimus completed their work and notified BizTalk via an HTTP receive, the orchestrations completed under the old version. New requests that were received via our EDAF service facade were then processed in the new version of the orchestration. Eventually, we'll be able to shut down the first version of the orchestration once all the existing Ultimus incidents have been resolved.

We also learned that with the addition of new orchestrations and a new service we need to automate the installation of all components using msi packages. We used BizTalk scripts which worked well but ensuring that the GAC was updated correctly on our services machines along with configuration files made the deployment to five application and four database servers a two and half hour process that included a couple missed configuration settings.

Architecturally the biggest challenge was in putting in place an "uber" orchestration that acts as the controller for all requests that are processed through our infrastructure to go along with a tracking database. This design will allow us to plug in new transaction types as we automate subsequent business processes.

Other interesting aspects included building a web services operation that attempts to match sponsors to unvalidated information submitted to the process (a so called "Professed" request as opposed to a "Validated" one where the user has provided credentials) and called from an orchestration responsible for "reconciling" sponsor data, and (through some slick .NET coding) allowing the Ultimus user interface to act as a consumer of our services using our Service Agent framework in order to retrieve and create sponsors.

Overall since our first release in September the service-oriented infrastructure has handled over 400,000 service requests and has been remarkably stable. I've been impressed with the EDAF infrastructure on which the services are based and BizTalk has easily handled the load we've thrown at it.

TDD and Visual Studio Team System

2005-11-22T11:14:00.000-08:00

A co-worker brought this post by Scott Bellware to my attention on the guidelines Microsoft has on MSDN for test-driven development and how these don't really conform to traditional TDD approaches.

Scott makes some excellent points the most important of which IMHO are:

TDD helps you design the code by writing the test first and then making the code conform through refactoring of the interfaces etc.

TDD tests don’t necessarily map one to one with methods in a class

This point about test generation in the tool came up at a developer event on Team System I attended last July in Redmond. I got the impression that the MSFT folks fully understood the concerns of the agile community (they were loudly voiced in much the same way Scott articulated in the blog) but that since not everyone was or is doing agile, having the autogeneration feature was still valuable. I agree that the guidelines miss the mark but that said there is a disclaimer in the document that they’re not following "traditional TDD".

I could see an organization using a mix of both approaches based on the type of development you're doing. When designing a class library from scratch for use by other developers I might following the "traditional" route so that I'm not constrained in my design by my first thoughts, but when another developer uses my library and inherits from it they might like to autogenerate some unit tests. They are more constrained and so using the autogeneration might be more productive for them.

We have this situation with the Service Agent Framework we built. I haven't spent that much time in VS2005 to know how I would use the Class Designer in the former approach but my guess is that as I refactor to conform the design to the test I’ll do most of that work in the Class Designer.

Messaging Standards

2005-10-07T06:09:00.000-07:00

Another aspect of the Service Communication part of our services platform are the messaging standards. These include both the envelope standards used for encapsulating messages as well as the message headers we use.

Envelope Standard
This envelope standard defines the XML schema used to represent messages. Rather than create a new standard the CI-SOA will use the Simple Object Access Protocol (SOAP) 1.1 standard envelope defined at http://www.w3.org/TR/2000/NOTE-soap-20000508/. As an example a request message looks as follows:


<soap:Envelope
 xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
 soap:encodingStyle=
   "http://schemas.xmlsoap.org/soap/encoding/"/>
   <soap:Header>
       <t:Transaction>5</t:Transaction>
   </soap:Header>
   <soap:Body>
       <m:GetLastTradePrice xmlns:m="Some-URI">
           <symbol>DEF</symbol>
       </m:GetLastTradePrice>
   </soap:Body>
</soap:Envelope>

This standard is intended to be used each time a service provider exchanges a message with a service consumer regardless of the binding used to call the service. For example, the same envelope will be used when invoking services over HTTP or MSMQ. As a result, our service agent framework builds SOAP v1.1 envelopes when services are invoked.

Further, developers use document style encoding (as opposed to RPC-style encoding) with XML Schema (XSD) when possible since doing so makes it possible to support arbitrarily complex types in XML messages. The .NET Framework supports document style encoding by default.

The reason the SOAP 1.1 standard is being used rather than SOAP 1.2 is that SOAP 1.1 is supported natively in the .NET Framework v1.1.

Header Standard
This header standard defines the XML schema used to add informational content to the Header element of the envelope. This information is useful for service management, routing, and security.

The standard includes two parts. The first is the WS-Addressing schema found at http://schemas.xmlsoap.org/ws/2004/08/addressing/. Elements in this schema are used for the following purposes.

Element Usage

MessageID (Required)- A GUID that uniquely identifies this message. Useful for logging and duplicate message handling

To (Required) The physical address the message is being sent to. Useful for routing. A URL for HTTP requests or a direct format name for a queue in the form of a URN, e.g. urn:DIRECT=OS:compassion.com\PUBLIC\MyQueue;JOURNAL

From (Required) The service or application that created the message. Typically uses only the Address element formatted as a URN to identify the creator of the message. Useful for logging.

Action (Required) The soapAction for the operation being invoked

ReplyTo (Optional) Contains an address used to send a response to. This can take the form of a URL for HTTP or a direct format queue name in the form of a URN. Optionally, this element may include reference properties if additional information is required for a particular endpoint. If not specified the reply is sent to the URI from which the original was sent. Services may also use this element in order to send delayed responses to a particular location. For example, the immediate response to the message may include an acknowledgement while the eventual complete response is sent to the ReplyTo address

RelatesTo (Optional) Populated in a response message from the request-response message exchange. Contains the MessageID of the original request message

The second schema is the http://schemas.compassion.com/common/headers/2005-04-05 schema that defines elements used within the SOAP Header element as well as the From element of WS-Addressing. The definition of the headers schema is as follows:


<?xml version="1.0" encoding="utf-8" ?>
<xs:schema id="CIHeaders" 
targetNamespace=
 "http://schemas.compassion.org/common/headers/2005-04-05"
 elementFormDefault="qualified"
 xmlns:ch=
 "http://schemas.compassion.com/common/headers/2005-04-05"
 xmlns:xs="http://www.w3.org/2001/XMLSchema">
 <xs:element name="MesssageStreamID" type="xs:anyURI"> 
      </xs:element>
 <xs:element name="CreatorIdentity" 
   type="xs:string"></xs:element>
 <xs:element name="DateTimeStamp" 
   type="xs:dateTime"></xs:element>
 <xs:element name="ProcessID" 
   type="xs:anyURI"></xs:element>
 <xs:element name="Culture" 
   type="xs:string"></xs:element>
</xs:schema>

The usage of the elements is as follows:

Element Usage

MesssageStreamID (Required) A GUID that identifier a unique sequence of messages that flow through the SOA. Used for correlating messages and for logging purposes. If the creator of the message does not have a MessageStreamID in hand from a previous message, it should create one and place it in the Header.

CreatorIdentity (Required) The windows account representing the creator of the message. Useful for logging and to authorize requests using the trusted subsystem model

DateTimeStamp (Required) Identifies in Coordinated Universal Time (UTC) when the message was created by the service consumer or provider. Useful for logging

ProcessID (Optional) A GUID that identifies a process. Typically populated by process services. If a service receives a ProcessID it should include it in subsequent messages

UICulture (Optional) The culture setting for the client specifying the language to use that follows RFC 1766 derived from ISO 639-1 and ISO 3166, e.g. US English="en-US"

In the future we'll likely also add a PartnerCountryCode element that identifies the Compassion partner country from which the request originated. This will allow services to take different actions (for example processing different business rules) when the requests come from different countries.

Examples
The following is an example request message that employs the standards discussed in this document:


<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"  
 xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
 xmlns:ch=
"http://schemas.compassion.com/common/headers/2005-04-05" 
 xmlns:wsa=
"http://schemas.xmlsoap.org/ws/2004/08/addressing">    
  <soap:Header>
   <ch:MessageStreamID>uuid:bbbb-cccc-dddd-eeee
   </ch:MessageStreamID> 
   <ch:DateTimeStamp>2005-04-05:T18:00:00
   </ch:DateTimeStamp> 
   <ch:CreatorIdentity>ci/dfox</ch:CreatorIdentity> 
   <ch:Culture>en-US</ch:Culture> 
   <ch:UICulture>en-US</ch:UICulture>
   <wsa:MessageID>uuid:aaaa-cccc-dddd-eeee
   </wsa:MessageID> 
   <wsa:To>http://compassion.com/DemoService.asmx
   </wsa:To>
   <wsa:Action>http://compassion.com/DemoServiceURI
   </wsa:Action> 
   <wsa:From>
     <wsa:Address>urn:CompassionWeb</wsa:Address>
   </wsa:From>
  </soap:Header>
  <soap:Body>
    <HelloWorld 
     xmlns="http://compassion.com/DemoServiceURI" />
  </soap:Body>
</soap:Envelope>

Note that the MessageStreamID is created by the service consumer since this is the first message in the stream. The CreatorIdentity is the Windows account of the creator of the message and the culture is the culture setting on the client machine making the request. In this case the message will pass over HTTP and so the To element contains the .asmx address and the Action element the soapAction from the WSDL contract. The From element contains a URN that identifies the application that created the message. Since no ReplyTo element has been provided the reply if any will be sent to the URI from the which the message was received or that defined by the WSDL for the service.

The reply to the above message would look as follows:


<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"  
 xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
 xmlns:ch=
 "http://schemas.compassion.com/common/headers/2005-04-05" 
 xmlns:wsa=
 "http://schemas.xmlsoap.org/ws/2004/08/addressing">    
  <soap:Header>
 <ch:MessageStreamID>uuid:bbbb-cccc-dddd-eeee
 </ch:MessageStreamID> 
 <ch:DateTimeStamp>2005-04-05:T18:02:00
 </ch:DateTimeStamp> 
 <ch:CreatorIdentity>ci/serviceAccount
 </ch:CreatorIdentity> 
 <ch:Culture>en-US</ch:Culture>
 <ch:UICulture>en-US</ch:UICulture>
 <wsa:MessageID>uuid:gggg-cccc-dddd-eeee
 </wsa:MessageID> 
 <wsa:To>urn:CompassionWeb</wsa:To>
 <wsa:Action>urn:CompassionWeb/ResponseMessage
 </wsa:Action>  
 <wsa:From>
  <wsa:Address>http://compassion.com/DemoServiceURI
  </wsa:Address>
 </wsa:From>
 <wsa:RelatesTo>uuid:aaaa-cccc-dddd-eeee
 </wsa:RelatesTo>
   </soap:Header>
   <soap:Body>
    <HelloWorldResponse 
      xmlns="http://compassion.com/DemoServiceURI" />
   </soap:Body>
</soap:Envelope>

Here the MessageStreamID is copied from the request message and new DateTimeStamp and MessageID are created. The CreatorIdentity is now the account under which the service is executing. The To element contains the URN of the service consumer from the request message and the Action element includes an identifier that identifies the type of message being returned. The From element contains the URI of the service that returned the result and the RelatesTo element contains the MesssageID of the request message.

In the second example that follows the service consumer includes a ReplyTo element that alerts the service provider as to where the response should be sent. In this case it is an MSMQ queue. The request queue is also specified in the To element. Note that the Action element still identifies the operation being invoked.


<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"  
 xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
 xmlns:ch=
 "http://schemas.compassion.com/common/headers/2005-04-05" 
 xmlns:wsa=
 "http://schemas.xmlsoap.org/ws/2004/08/addressing">    
 <soap:Header>
 <ch:MessageStreamID>uuid:bbbb-cccc-dddd-eeee
 </ch:MessageStreamID> 
 <ch:DateTimeStamp>2005-04-05:T18:00:00</ch:DateTimeStamp> 
 <ch:CreatorIdentity>ci/dfox</ch:CreatorIdentity>
 <wsa:MessageID>uuid:aaaa-cccc-dddd-eeee</wsa:MessageID>
 <wsa:To>urn:DIRECT=OS:compassion.com\PUBLIC\RequestQ
 </wsa:To>
 <wsa:Action>http://compassion.com/DemoServiceURI
 </wsa:Action>         
 <wsa:ReplyTo>              
   <wsa:Address>
     urn:DIRECT=OS:compassion.com\PUBLIC\ResponseQ        
  </wsa:Address>
 </wsa:ReplyTo>
 <wsa:From>
  <wsa:Address>urn:CompassionWeb</wsa:Address>
 </wsa:From>
   </soap:Header>
  <soap:Body>
    <HelloWorld 
     xmlns="http://compassion.com/DemoServiceURI" />
  </soap:Body>
</soap:Envelope>

Schema Standards

2005-10-06T06:08:00.000-07:00

One of the aspects of our services platform is service communication which I discussed at a high level awhile back. One of the detailed aspects of this area is related to how schemas are defined for our contracts.

Obviously, we're using XSD to define the schemas and we chose to build all the schemas within BizTalk. The reasons behind this are twofold: we wanted to standardize on a tool for building the schemas and we wanted to make sure the message contracts we defined would work in BizTalk since it is the central hub for all of our long running process services.

What follows is a discussion of the related standards we used for creating schemas for use in the Compassion SOA that a namespace standard, a naming standard, a versioning standard, and an extensibility standard.

Namespace Standard
This namespace standard defines how namespaces will be used within the schemas produced for the Compassion SOA.

The namespace used to identify schemas will follow a three part convention.

Part I: A reference to http://schemas.compassion.com. All lowercase should be used.

Part II. A domain specific identifier. For example constituent, common, child, etc. If there are sub-domains that need to be specified, use a backward slash as in constituent/profile. All lowercase should be used.

Part III. The date on which the schema was finalized. The data is meant to uniquely identify this version of the schema. When a breaking change is made to the schema, a new schema should be created with a new date. Dates should be in the form yyyy-mm-dd, e.g. 2005-04-05.

Note: A breaking change to the schema is defined as one where non-optional elements or attributes are added to the schema, where data types are changed, or where the document structure is incompatible with the previous version.

Valid domain names that have been identified at this time include:

constituent – includes constituent management schemas

common – includes common schemas such as headers, exceptions, and workflow

child – includes child management schemas

Examples using this three part convention include:


http://schemas.compassion.com/constituent/2005-03-28
http://schemas.compassion.com/constituent/2005-05-05 
http://schemas.compassion.com/common/headers/2005-04-05  
http://schemas.compassion.com/constituent/profile/2005-03-28

Naming StandardThe naming standard is in place in order to specify how elements and attributes are named. It consists of the following:

Element names should use Pascal casing (upper case first letter of each word)

Attribute names should use Camel casing (lowercase first letter of the first word, upper case first letter of all other words

Attributes should be used for identifying information within an element. For example identifiers such as ConstituentID would be modeled as an attribute

The ID suffix should be uppercase when used

Versioning Standard
The versioning standard is in place in order to support visibility to consumers of documents produced using the schema for non-breaking changes. For example, if a schema includes a new optional element or attribute, the versioning standard allows a consumer of the schema to note the version in order to dynamically add the appropriate element.

The versioning standard mandates the inclusion of the following attribute in the root node of the schema:


<xs:attribute name="schemaVersion" type="xs:decimal" />

By adding this attribute documents can be created that specify the version number in addition to the unique identifier specified in the namespace.

When a new schema is created, documents created using the schema should default the schemaVersion to 1.0. As revisions are published the latest schemaVersion will be placed in instance documents, for example, 1.2. Since the version numbers are not visible within the schemas but only in the instance documents, the schema repository will track older versions of the schema.

Extensibility Standard
In order to provide extensibility for consumers of schemas as the schema evolves the following extensibility standard has been created. The following element should be added to the root element of the schema as well as any other elements where extensibility may be expected.


<xsd:any namespace="##any" processContents="lax"
minOccurs="0" maxOccurs="unbounded"/>

This element insures that additional elements of any number from any namespace may occur within this element and that a consumer is not obligated to validate the contents ("lax").

The second aspect of this standard is that the root element should also contain the following element.


<xs:anyAttribute namespace="##any" processContents="lax"/>

This allows additional attributes to be placed in the root element. This standard has implications for the way in which .NET schema consumers behave. Using this standard, .NET classes created using the Xml Schemas/DataTypes Support Utility (xsd.exe) will include arrays of XmlElement and XmlAttribute objects that are decorated with the XmlAnyElementAttribute and the XmlAnyAttributeAttribute. To populate these elements consumers will then need to create these types and add them to the array. While this provides extensibility, the extension objects are not strongly typed. See the examples section for the sample code to do this.

Examples
An example schema follows:


<?xml version="1.0"?>
<xs:schema 
 xmlns=
  "http://schemas.compassion.com/constituent/2005-03-01"
 targetNamespace=
   "http://schemas.compassion.com/constituent/2005-03-01"
 xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Constituent">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Name" type="xs:string" 
          minOccurs ="1" />
        <xs:element name="Address1" type="xs:string" 
          minOccurs ="1" />
        <xs:any namespace="##any" processContents="lax" 
           minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="schemaVersion" 
        type="xs:decimal" />
      <xs:attribute name="ConstituentId" 
        type="xs:long" />
      <xs:anyAttribute namespace="##any" 
        processContents="lax" />
    </xs:complexType>
  </xs:element>
</xs:schema>

Note the presence of schemaVersion as well as the namespace that adheres to the Namespace Standards defined above and the extensibility points.

An instance document using this schema would then look as follows.


<?xml version="1.0"?>
<Constituent>
  xmlns=
   "http://schemas.compassion.com/constituent/2005-03-01"
  ConstituentId="1234" schemaVersion="1.0">
  <Name>John Jones</Name>
  <Address1>555 Elm Street</Address1>
</Constituent>

If a breaking change occurs, for example, an Address2 element is added that is not optional (it’s minOccurs attribute is set to "1") then the new schema would be defined as follows.


<?xml version="1.0"?>
<xs:schema 
 xmlns=
  "http://schemas.compassion.com/constituent/2005-04-01"
 targetNamespace=
"http://schemas.compassion.com/constituent/2005-04-01"
 xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Constituent">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Name" type="xs:string" 
          minOccurs ="1" />
        <xs:element name="Address1" type="xs:string" 
          minOccurs ="1" />
        <xs:element name="Address2" type="xs:string" 
          minOccurs ="1" />
        <xsd:any namespace="##any" processContents="lax" 
           minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="schemaVersion" 
        type="xs:decimal" />
      <xs:attribute name="ConstituentId" 
        type="xs:long" />
      <xsd:anyAttribute namespace="##any" 
        processContents="lax" />
    </xs:complexType>
  </xs:element>
</xs:schema>

Note that the targetNamespace has been changed since the schema contains a new element.

However, if the schema published under the http://schemas.compassion.com/constituent/2005-03-01 namespace is modified to support an optional Address2 (minOccurs = "0"), then documents created using the new version should include the incremented schemaVersion, for example, to 1.5 as in the following example document.


<?xml version="1.0"?>
<Constituent> 
  xmlns=
   "http://schemas.compassion.com/constituent/2005-03-01"
  ConstituentId="1234" schemaVersion="1.5">
  <Name>John Jones</Name>
  <Address1>555 Elm Street</Address1>
  <Address2>Apt 5678</Address2>
</Constituent>

In this way consumers of the new document will be valid under the original schema (1.0) through the extensibility points and the document can be identified as varying from the original through the schemaVersion. Documents created under the original version of the schema (1.0) will be valid under the new version since Address2 is optional.

In order to illustrate how the extensibility portion of the standard works consider the following code.


<XmlTypeAttribute([Namespace]:= _
"http://schemas.compassion.com/constituent/2005-03-01"),  _ 
   XmlRootAttribute([Namespace]:= _
"http://schemas.compassion.com/constituent/2005-03-01", _ 
  IsNullable:=false)>  _
Public Class Constituent 
    <XmlElementAttribute()>  _
    Public Name As String

    <XmlElementAttribute()>  _
    Public Address1 As String

    <XmlAnyElementAttribute()>  _
    Public Any() As XmlElement

    <XmlAttributeAttribute()>  _
    Public schemaVersion As Decimal

    <XmlIgnoreAttribute()>  _
    Public schemaVersionSpecified As Boolean
    <XmlAttributeAttribute()>  _
    Public ConstituentId As Long

    <XmlIgnoreAttribute()>  _
    Public ConstituentIdSpecified As Boolean

    <XmlAnyAttributeAttribute()>  _
    Public AnyAttr() As XmlAttribute
End Class

This code was generated using the schema shown previously using the xsd.exe utility. As you can see the tool generated an array of XmlElement objects called Any and an array of XmlAttribute objects called AnyAttr. Client code that needs to populate the new version of the schema (1.5) with the optional Address2 element would then need to do the following.


Dim c As New Constituent

' Populate the document
c.Name = "Willie Mays"
c.Address1 = "Candlestick Park"
c.ConstituentId = "1234"
c.schemaVersion = 1.5

' Create the Address2 element
Dim xdoc As New XmlDocument
Dim x1 As XmlElement = xdoc.CreateElement("Address2", _
"http://schemas.compassion.com/constituent/2005-03-01")
Dim x2 As XmlText = 
 xdoc.CreateTextNode("San Francisco Giants")
Console.WriteLine(x1.NamespaceURI)

' Append it to the document
x1.AppendChild(x2)

' Add it to the Constituent
Dim ext(0) As XmlElement
ext(0) = x1
c.Any = ext

The disadvantage to this approach this approach is that the code required to add the additional element is fairly cumbersome. However, doing so allows the Constituent class to remain unchanged in the event that it resides in a different assembly that will not be recompiled and redistributed to clients.

Too Small To Ignore

2005-10-06T05:30:00.000-07:00

Yesterday the President and CEO of Compassion Wes Stafford spoke to the entire organization about his new book Too Small to Ignore: Why Children are the Next Big Thing published by Waterbrook Press.

The book is a wakeup call that brings home the message that children matter and that those who have the power (which ultimately includes all of us) should use that power to be the voice for improverished children around the world. Wes grew up in rural west Africa (Ivory Coast) as the son of missionaries and lived among the people that Compassion now serves. He has a unique insight into the problem and the face of poverty that he also shares in the book in recounting some of his journey.

The book will hit stores October 18th but you can preorder from the link above.

Transaction Handling in EDAF

2005-10-04T14:24:00.000-07:00

In my previous post I mentioned that I'd share a bit about how we modified EDAF v1.1 for our services platform here at Compassion. There are several ways we've done this that include:

Modifying the service interface adapters to use our messaging standards

Modifying the exception reporting framework to log exceptions to our central logging store

Creating a base class for business actions to include common code

Modifying the TransactionHandler to support a model where business actions do not throw exceptions and transactions can span the invocations of business actions

It is this final one that I'll explore today, the others will have to wait for a future post.

Background
There have been two major efforts at creating web service specifications in the industry as documented by Newcomer and Lowow in chapter 10:

WS-Transactions. This includes a family of specifications developed by Microsoft, BEA, and IBM that includes WS-AtomicTransaction, WS-BusinessActivity, and WS-Coordination.

WS-Composite Application Framework. This is a family of specifications developed by Arjuna, IONA, Oracle, Fujitsu, and Sun and includes WS-Context, WS-CoordinationFramework, and WS-TransactionManagement.

These specifications are similar and have considerable overlap. For example, in both cases "participants in a transaction register with a coordinator and specify the protocol type so that the coordinator can drive the appropriate protocol for use." Both can support various transaction processing requirements including two-phase commit, compensation, and business process transactions. The hope is that these will merge into a single specification in the future.

However, in our evaluation of these specifications it was apparent that there is no infrastructure in place to support either. In other words, there would need to be a set of code in place that interpreted the SOAP header information specified in these various specifications and performed the registration as well as controlled the transaction. Support for WS-Transaction will be built into Indigo within Win/FX, however.

Approach
Because it is not practical for Compassion to implement the kind of infrastructure required, the approach we’ve taken is to modify and use the Transaction Handler present in EDAF to implement Component Services transactions across Business Actions.

The Transaction Handler within EDAF takes advantage of EnterpriseServices (COM+) in the .NET Framework to implement transaction support. The handler itself uses an object named TransactionRequired, which is registered with COM+ by the TransactionHandlerInstaller assembly. The TransactionRequired object is configured to require transactions and set the isolation level to Serializable. Since this is a stacked around handler the target will always be included in the transaction. In addition, any handlers that are wrapped by this handler will also be included in the transaction. If execution is successful, the transaction is committed. If an exception is thrown, then the transaction is stopped. This handler can then be placed within the Service Implementation pipeline

In our architecture services report both business and system exceptions within an Exceptions element that is returned within the SOAP body as in the example below.


<Exceptions 
  xmlns="http://schemas.compassion.com/common/
    exceptions/2005-03-01">
  <Exception Type="System" xmlns="">
     <DateTimestamp>0001-01-01T00:00:00.0000000-07:00
     </DateTimestamp>
     <Code>Compassion.Common.Data</Code>
     <Message>Could not create data reader from 
      statement GetCon</Message>
     <StackTrace> at Compassion.Common.Data.
      DataFactory._throwException…
     </StackTrace>
  </Exception>
  <Exception Type="Business" xmlns="">
      <DateTimestamp>2005-05-18T09:09:54.9242939-06:00
      </DateTimestamp>
           <Source>Execute</Source>
           <Code>INVALID_ID</Code>
           <Message>Constituent with ID [353594] 
              was not found.
           </Message>
           <StackTrace> at Compassion.Services.
              Actions.Constituent…
           </StackTrace>
   </Exception>
</Exceptions>

As a result, the Business Actions we develop will not throw .NET exceptions to their callers and in turn when using the TransactionHandler transactions will not typically be rolled back since the TransactionRequired class is marked with an AutoComplete attribute and so will only rollback when an exception is thrown from within one of the business actions or handlers it is encapsulating.

To address this situation we’ve modified the TransactionRequired class within EDAF to remove the AutoComplete attribute and instead rollback or commit transactions using the ContextUtil.SetAbort and ContextUtil.SetComplete methods respectively. The class will do this when it finds that the TransactionValid key (a key the handler itself initializes) in the EDAF Context is set to false. This technique is analogous to the way in which COM+ manages transactions internally. Each component (in this case business action) that participates in the transaction "votes" on its outcome by either setting the flag to false signifying that the business action is not happy and the transaction should be rolled back or doing nothing which signifies that the business action is happy and the transaction should continue.

Transaction context will flow across invocations of business actions since the TransactionRequired will have its transactional support set to Required as shown in the following dialog.

In this way, the first business action that is invoked will act as the root of the distributed transaction. Each subsequent business action will be wrapped in another instance of the Transaction Handler and will enlist in the already started transaction. If one of the nested business actions sets its TransactionValid flag to false, its Transaction Handler will call ContextUtil.SetAbort thereby aborting the entire distributed transaction and rolling back the work done for all business actions in the call stack. This is illustrated in the following diagram where the step numbers denote the timing.

Implications
The implications of this technique are threefold:

First, developers of business actions are required to set the TransactionValid flag to false when they wish their database changes to be rolled back. The general pattern for doing so is shown below.


public void Execute(IContext objContext)
{
 try
 {
  // Perform database writes or call
  // other business actions
 }
 catch (Exception ex)
 {
  // Handle System Exceptions and create Exception 
  // elements to add to the response
 }
 finally
 {
  // Inspect response and create business Exception
  // elements to add to the response
 }

// Vote on the outcome of the transaction
 if (objResp.Exceptions.Length>0)
  if (objContext["TransactionValid"] != null)
objContext["TransactionValid"] == false;
}

Here, the entire activity of the business action is encapsulated in a try/catch block. System and business exceptions are detected either through catching exceptions thrown from the data access layer or by inspecting the contents of the response returned from the data access layer.

It should be noted that in our implementation we've abstracted the setting of the TransactionValid flag in a base class from which all business action classes inherit.

Using this approach implies that the business action will be implemented within a Service Implementation pipeline in which the Transaction Handler is configured. The work done in the try block will then be rolled back when the Transaction Handler aborts the transaction. Therefore business actions should not use local transactional control (they should not be starting and committing local transactions).

Secondly, this architecture implies that in order for the transaction context to flow across business action requests the business action must be invoked using the in-process service interface adapter. In other words, if business action A invokes business action B and the work from the both actions needs to be part of a single distributed transaction, action A must invoke action B using the in-process adapter. This is the case since distributed transaction cannot flow across the other interface adapters in EDAF which include web services, MSMQ, and .NET Remoting.

For example, a business action could invoke the SelectEmailRequest business action using the in process adapter like so:


InProcDispatchingAdapter objDA = 
  new InProcDispatchingAdapter();
Request objRequest = new Request();
objRequest.ServiceActionName = "UpdateAddress";

objRequest.Payload = 
  XmlSerializerHelper.SerializeToXml(objUpdateRequest);

Context objDAContext = new Context(objRequest);
objDAContext.Add("ValidateOnly",this.ValidateOnly);

objDA.Submit(objDAContext);

//get the response.
AddressUpdateResponse objCompletedResponse = 
  (AddressUpdateResponse)
  XmlSerializerHelper.Deserialize(((XmlDocument)
  objDAContext.Response.Payload).InnerXml,
   typeof(AddressUpdateResponse));

//append any exceptions.
this.AppendExceptions(objCompletedResponse.Exceptions);

Here you’ll notice that a new Context object is created for the request to the UpdateAddress business action. In this case each business action will have its own Context object in which the Transaction Handler will check for and use the TransactionValid flag.

The third implication of this approach is that service consumers need to be aware that there is no way in our current architecture of flowing transaction context across two or more service invocations. For example, if a service consumer uses a service agent to invoke the UpdateConstituent operation and then uses the same service agent to invoke CreateConstituent, these two invocations cannot participate in the same distributed transaction. This is the case since the service agents will invoke the services using SOAP over HTTP or MSMQ, neither of which offer distributed transactions at present. The thought is that in the future support for transactions across service calls will be supported by the Indigo framework in Win/FX using WS-Transactions, which can then be integrated into the service agent framework.

Selecting EDAF

2005-09-19T14:07:00.000-07:00

When I left off at the end of May I was discussing EDAF and how we used it to implement our services. What follows is a description of EDAF and how we use it.

In order to implement Activity (some of them), Entity, and Infrastructure Services we've taken the approach of using the guidance created by the Microsoft Patterns and Practices Group (PAG). The collection of Microsoft best practices in this area comes in the form of the Enterprise Development Reference Architecture (EDRA) formerly known as "Shadowfax". This .NET code base is designed to provide architectural guidance to organizations moving to service orientation by including the following:

An extensible application framework called the Enterprise Development Application Framework (EDAF)
Four QuickStart applications to familiarization with EDRA
An application template and supporting documentation used to build new services
The Global Bank Reference Implementation (GBRI) that uses EDRA to implement an online banking scenario
A good base of documentation on which to work

All of the information for EDRA including its releases can be found on its GotDotNet workspace at http://workspaces.gotdotnet.com/shadowfx. In addition, more information can be found on the Channel 9 EDRA Wicki at http://channel9.msdn.com/wiki/default.aspx/Channel9.EDRAWiki.

For the CI-SOA (Compassion International SOA) the final released version, 1.1, is being used. In addition the CI-SOA will integrate several tools included with EDRA in the future.

UI support for Visual Studio
A service agent
A configuration editing tool

At a high level, the EDAF has four primary goals:

Separating the service interface from the service implementation
Separating the business logic from the cross-cutting concerns mentioned previously
Separating the business logic from the underlying transport so that different transports can be used to access a service
Develop a stable service interface to ensure resiliency when services are deployed

In evaluating whether or not the EDAF would be used for the CI-SOA the project, the other options included:

Implementing Entity and Infrastructure services as simple .ASMX. This is the approach Microsoft recommends in order to make it as simple as possible to move to the new Indigo service framework when it is released (for example See David Chappell’s overview of Indigo at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnlong/html/introindigov1-0.asp) in the spring of 2006. Specifically, the guidance offered was in opposition to implementing services today using Enterprise Services or .NET Remoting. Using straightforward .ASMX, however, does not provide the infrastructure for duplicate message handling (idempotent messaging), message logging, performance monitoring, access through multiple interfaces, and other cross-cutting concerns. Since these are standards in our Services Platform that the project team will implement, and since it was not practical to wait for Indigo in order to implement since the first deliverable in August 2005, it was decided that the EDAF provided the core framework needed to build services. The thought is that the separation of service interface from service implementation that the EDAF provides will allow the business logic of Compassion services to be re-factored into Indigo services when it is deemed practical in the future.
Implementing Entity and Infrastructure services as BizTalk orchestrations. The other option that was considered was implementing all services as BizTalk Orchestrations and exposing those services using the BizTalk orchestration web publishing wizard as illustrated in the following diagram.

The advantages of doing so include:

The fact that some of the cross-cutting concerns such as logging and performance monitoring are already implemented within BizTalk Server
There is expertise in the industry regarding BizTalk and how to build orchestrations along with examples and schemas
It provides a one-stop approach to upgrading
Future interoperability with technologies such as Indigo will be packaged into upgrades of BizTalk server.

The disadvantages of doing so include:

A complete dependency on BizTalk for all service development which reduces flexibility
The need to build façade type orchestrations that may become complex in order to expose multiple operations in a single web service
The ability to perform maintenance and upgrade pieces of the architecture is reduced
The need to build out other cross-cutting concerns within BizTalk including duplicate message handling, and authorization
The need to roll out a set of services in a Partner Country or Field Office requires that BizTalk server be deployed and maintained in that environment. Services built using EDAF would require a smaller footprint and investment in terms of licensing. At present, however, there is no small-footprint option for deploying process services. We assume that if those require orchestrations BizTalk would be installed in the field.

In final analysis, the decision was made to take advantage of the strengths of BizTalk which include orchestrating long running business processes and not attempt to shoe-horn the implementation of more straightforward request-response style services into it. The only caveat to the above is that Activity Services that require transactional coordination among Entity Services or the ability to retry operations in the event of a data store being unavailable will be implemented as BizTalk orchestrations. The reason is that transactional coordination across service invocations is not implemented in the standard platform shipped by Microsoft (for example through support of WS-AtomicTransaction) nor is it a part of the EDAF (the EDAF supports compensating business actions and atomic transactions within a service invocation but not across service invocations). It will be a part of Indigo, however, and so Activity Services should be able to be implemented using Indigo in the future.

In order to satisfy these goals the EDAF implements two primary patterns.

Interception Pattern
The EDAF abstracts how the service is invoked (its service implementation) from the core logic of the service (its service implementation) as shown below. This allows services to be invoked in a variety of scenarios, for example by web sites, Windows applications, Smart Device applications, and even by a BizTalk orchestration that places the message in a queue. This architecture also allows organizations to customize their deployment across several severs although typically the service interface will use the Web Services interface transport and the service implementation the InProc dispatching transport.

The space between the client and the service interface and the service interface and service implementation includes the notion of transports and adapters. The transports abstract the invocation of the service (interface transports) and its implementation (dispatching transports) while the adapters are responsible for receiving the message and initiating message processing.

The interface transports that ship with EDAF include Web Services, .NET Remoting, and Message Queuing. The dispatching transports include Web Services, Message Queueing, DCOM, and in-process (InProc). Each transport, with the exception of InProc needs a host and so for web services the host is IIS/ASP.NET. For .NET Remoting and MSMQ the EDAF provides both a Windows Service called TransportService.exe and a console application called TransportConsole.exe.

Organizations can also implement their own interface transport by creating a class that implements the ITransportListenerController interface and hosting it the provided hosts or an executable. This interface includes simple Start and Stop methods that are called by the EDAF to begin and to quit listening for requests respectively. For example, one could create an FTP Transport by hosting a component in a Windows Service that "listens" for requests by polling the folder used by an FTP site.

Each of the interface transports has an associated adapter that is responsible for extracting the request message from the transport and creating the EDAF Context object which is a kind of wrapper object that contains all of the information needed to process the request. The Context is then passed into the service interface. Likewise, each dispatching transport has an adapter that retrieves the Context object and passes it into the service implementation. Each adapter is customized for its transport, for example, the Web Services interface transport’s adapter is implemented as a SOAP extension.

This pattern is an integral part of EDAF and will be used when implementing services in order to separate the service interfaces from the service implementations. By doing so the service implementations can be invoked from a variety of service interfaces.

Pipeline Pattern
One of the major goals of the EDAF is to separate cross-cutting concerns such as logging, security, transactional behavior, and event notification from the business logic. To achieve this goal both the service interface and the service implementation include the concept of pipelines. A pipeline consists of a chain of handlers that each implement one of the cross-cutting functionalities as shown below. This is an implementation of the well-known Chain of Responsibility pattern.

These pipelines are executed by the adapters and passed the Context object discussed in the previous section. When all of the handlers have been executed the pipeline finally executes a target object. The target object is specific to what it will execute, for example, whether the target is to pass the Context object to the dispatching transport or whether to a business action.

The service interface pipeline shown in the diagram is transport-specific and typically focuses on boundary concerns such as authentication, request monitoring, validating the request message, and handling duplicate request messages by returning a cached result (idempotence). As a result, the service interface pipeline might be configured to include the AuthenticationInfrastructure and AuthorizationInfrastructure handlers to perform client authentication and authorization using the Windows identity and either Active Directory or NTLM groups.

The pipelines and the handlers that are included are configured in the EDAF configuration file making these easy to adjust and turn on and off as needed.

The service implementation handlers on the other hand usually include service specific behaviors such as raising business events using WMI, logging requests, implementing transactions, or transforming messages. The following table shows the handlers that ship with the EDAF.

Handler
ClientTrace                  
Returns a trace of the execution of the request with the response

AuthorizationDatabase        
Performs authorization of the request using a SQL database

DuplicateHandling            
Implements once-only processing of a message

AppInstrumentationExecution  
Monitors the time taken to execute the request from this point forward

AuthenticationIdentity       
Trusts the current credentials passed in with the request

AuthenticationInfrastructure 
Ensures that the identity is authenticated using Windows

AuthorizationInfrastructure  
Authorizes a request using Windows groups

AppInstrumentation           
Updates performance monitor counters

MessageTransformation        
Uses an XSLT to transform a message

PasswordAuthorizationHandler
Performs user name and password authentication

PublishBusinessEvent         
Publishes a business event

SignedMessageAuthentication  
Uses a public key to verify the signature of an XML message

SyntacticValidation          
Transforms the message into a strongly-typed message

ExecutionTimeout             
Monitors the execution of the request beyond this handler and times it

Transaction                  
Starts a new COM+ transaction

Handlers also provide an extensibility point where the project team will insert custom processing in the pipeline. The EDAF includes three handlers interfaces that include IAtomicHandler for basic process, IStatefulAroundHandler for handlers that need to execute in the same stack frame, and IStatelessAroundHandler for handlers that execute both before and after the target is executed.

This pattern is an integral part of EDAF and will be used to add handlers that participate in the processing of service requests and responses. The cross-cutting concerns mentioned above are implemented with handlers.

In the next post I'll take a look at how we customized EDAF to fit our Services Platform.

Real SOA!

2005-09-19T13:41:00.000-07:00

I know it's been a long time since I published an update on our SOA implementation here at Compassion International. However, it's been a busy summer (and I've been doing alot of baseball writing as well). I'm pleased to report though that on August 30 we rolled out our first deliverable based on this architecture. This deliverable helps to automate the updating of sponsor information entered on our web site and to kick certain requests to a human workflow process for resolution based on a series of business rules.

The solution consists of our external web site Compassion.com, a web service with a series of web methods implemented using Microsoft's EDAF (about which more to follow) implemented in C#, MSMQ, three BizTalk Server 2004 orchestrations and two workflow processes implemented in the Ultimus BMP Suite 7.1.

Since it went into production we've recorded over 35,000 service calls and the system is humming along quite well. Currently, we're in the process of making a few tweaks in order to provide a 1.1 release before moving on to our next deliverable.

During this time I'll take the opportunity to continue the discussion of our Services Platform.

Implementing Services

2005-05-31T11:13:00.000-07:00

In my previous post I mentioned that we made a decision to use the Microsoft Development Application Framework (EDAF) when constructing our services in order to handle duplicate messages sent to service providers. While that is true, that's not the only reason we decided to use EDAF. I should add that we're not using EDAF for all the services in our taxonomy.

What we did decide was to implement the various types of services as follows:

Process Services. Implemented as BizTalk 2004 orchestrations or Ultimus workflows. These services will be exposed both through a Service Façade implemented as an EDAF operation (business action). A Process Service may encapsulate more than one orchestration if those orchestrations are a logical part of the process. However, if a Process Service invokes another Process Services it would do so through the Service Agent and its Service Façade. In other words logical boundaries need to be drawn between Process Services even if they are implemented as orchestrations on the same BizTalk installation.
Activity Services. Where transactional coordination of the entity services is required, implemented as BizTalk orchestrations. These services will be exposed through a service façade implemented as an EDAF operation. Where transactional coordination is not required, for example in the case where a business activity can be encapsulated in a single Domain Layer stored procedure that includes the transactional control, the service can be implemented as an EDAF services
Entity Services. Implemented as EDAF services
Infrastructure Services. Implemented as EDAF services

The way these services interact can be seen in the following diagram.

As mentioned above, our architecture calls for the implementation of Service Facades through which services of all types will be invoked. This is one of the two primary patterns that we wanted to implement and which EDAF could really help us tackle. More formerly, these patterns are:

Service Façade. This pattern implements services using a common framework to ensure consistent enforcement of a variety of operational requirements such as management, performance monitoring, security, and logging. In order to implement these cross-cutting concerns our architecture employs the Enterprise Development Application Framework (EDAF) version 1.1.
Service Agent. As discussed in my previous post this pattern encapsulates the concerns of service consumers with regard to service discovery, security credentials, and intelligent caching of time-sensitive data (e.g., token for a reservation message pattern). Because Compassion will primarily include clients built on the Microsoft .NET platform, the service agents will be developed as .NET class library assemblies

In the next post I'll lay out our justification for using EDAF.

Service Communication

2005-05-31T09:09:00.000-07:00

The next in the series of standards we published as a part of the our Services Platform is that of Service Communication. Essentially, this set of standards encompasses the means by which services communicate and includes the following requirements:

The ability to provide guaranteed messaging from a service consumer to a service provider
The ability to ensure that a service provider only processes a unique message one time, even if it is received multiple times (idempotent messaging)
The ability for every message to have a unique message ID in the header of the message
The ability to track how messages flow through the services providers to delegate (sub-contract) service providers
The ability for a service provider to know which service consumer called it

In short, the way we tackled these requirements was to implement the following:

MSMQ for guaranteed messaging, a part of which includes the WSDL extensions I talked about previously
The Microsoft Enterprise Development Application Framework (EDAF) to handle duplicate messages through its Duplicate Handler
The specification of a Compassion-specific SOAP header in combination with WS-Addressing to carry information such as the message ID, the relationship of messages to each other, and caller and callee information

In future posts I'll explain the last two of these in more detail.

Service Agent

2005-05-19T11:33:00.000-07:00

The third of the seven categories in our Services Platform is that of Service Agent. This pattern encapsulates the concerns of service consumers with regard to service discovery, security credentials, intelligent caching of time-sensitive data (e.g., token for a reservation message pattern), as well as invocations of the service of course.

The core tenets of how we use Service Agents in our platform is encapsulated in the statements below.

Service agents are required. Every .NET service consumer Compassion develops must employ a service agent. Therefore service developers must create a Service Agent for clients to use.
Service agents are client-side assembly. The Service Agent is deployed in a client-side .NET assembly other than the consuming service’s assembly
Service agents use a common interface. A service agent interface or base class is inherited by specific service agents. This enforces common behavior among service agents and allows for code reuse.
Service agents use helper classes. Service Agent Helper classes are general purpose utility classes that are invoked by the service agent and provides the implementation of specific functionality. For example, the communication with the service directory to implement service discovery is encapsulated in a helper class used by our service agent base class described below.

Essentially, when an application needs to use functionality provided in an external service, our service agent manages the semantics of communicating with that particular service. For example, the business components of a retail application could use a service agent to manage communication with the credit card authorization service, and use a second service agent to handle conversations with the courier service. Service agents isolate the idiosyncrasies of calling diverse services from applications, and can provide additional services, such as basic mapping between the format of the data exposed by the service and the format your application requires.

To assist with this standard a talented consultant, John McPherson of Interlink, created an elegant framework for building our service agents. For example, a ServiceAgentBase class inherits the following interface.

public interface IServiceAgent
{
XmlDocument ExecuteOperation(
string operation,
XmlDocument payload,
string messageId,
string messageStreamId,
string processId);

void ExecuteOperationNoResponse(
string operation,
XmlDocument payload,
string messageId,
string messageStreamId,
string processId);


string ApplicationName
{
get;
set;
}

string Type
{
get;
set;
}

string ReplyToAddress
{
get;
set;
}
}

As you can see our basic interface allows for request-response and one-way message exchange patterns through the ExecuteOperation and ExecuteOperationNoResponse methods. The service agent also allows for specifying the application name, communication type to use, and the address to reply to. You'll notice that these method signatures also include three ids - message, message stream, and process. These identifiers are placed into the header of the SOAP message so that our service management features are able to track how messages flow through our SOA. We've defined a standard SOAP header that I'll lay out in a future post.

The ServiceAgentBase class then exposes these methods as protected so that derived classes may use them when implementing specific methods that map to service operations. This makes the derived classes very light allowing us to implement methods that map to service operations using less than 25 lines of code. These derived service agent classes use specific schemas for request and response messages built using the XSD.exe tool that ships with the .NET Framework SDK. The schemas are originally built in BizTalk since our Process Services use them. We experimented with using the XSDObjGen tool to generate the "bind" classes but found it had several limitations when dealing with schemas with "any" elements and so we went back to XSD although it doesn't do as good a job of representing repeating elements since it doesn't use collection classes. In a future post I'll talk about our schema standards and how we plan to support extensibility.

We chose not to use the same technique (namely inheriting frmo SoapHttpClientProtocol) Visual Studio does when creating these service agents or proxies for a couple of reasons. First and foremost, our service agents allow for invoking services over several transports which include SOAP over HTTP and SOAP over MSMQ. The SoapHttpClientProtocol class obviously does not include support for MSMQ. Secondly, our service agents create and consume specific SOAP headers as mentioned previously.

The ServiceAgentBase class relies on a set of helper classes to interact with the service directory such as UDDIServiceDirectory that inherits the IServiceDirectory interface I discussed in a previous post. The ServiceAgentBase can then read configuration information using a configuration section handler class and either load and parse the service contract (WSDL 1.1) from a local path or query the service directory based on a service key (GUID) to download and parse the WSDL. The service agent relies on the WSDL to set the SOAP action and the endpoint at which to communicate with the service.

The framework also supports the idea of invoking services through a generic proxy class called ServiceAgentProxy. This class accepts payload XmlDocument objects and uses its configuration information to create and send the SOAP messages without the client having to explicitly reference assemblies that contain specific service agent classes. This functionality has been useful in two respects already. First, we use it when invoking services from inside BizTalk orchestrations. Within BizTalk we've created a generic orchestration that subscribes to messages that contain all the metadata about the call to make as well as the payload. It then uses the ServiceAgentProxy to make the call. Secondly, we're planning on using this approach to provide a way to invoke our services from ASP 3.0 code (which is a requirement in our first deliverable). Since we didn't want to use COM interop on the ASP server we created a Service Agent Broker ASP.NET site that accepts HTTP posts that contain the metadata about the call to make as well as the payload. The broker then uses ServiceAgentProxy to make the call and return the results to the ASP site using a Response.Write.

While we haven't added more sophisticated features we are planning on using service agents to do client side caching of reference data as well as using it to implement more complex message exchange patterns such as asynchronous request response using the reservation pattern.

Service Contract

2005-05-17T11:21:00.000-07:00

The second category of our Services Platform is Service Contract defined as "The request and (optional) response schemas for each operation exposed by a service, the security protocols, and the bindings associated with a service." Here we define "binding" as the combination of the transport protocol (e.g., HTTP, .NET remoting, MSMQ), the headers (WS-Security, WS-ReliableMessaging), and the encoding method (compressed binary, text, etc.). The W3C defines it as a concrete protocol and data format specification for a particular port type.

In looking at what was out there to define service contracts it should come as no surprise that we landed on Web Services Description Language (WSDL) 1.1. The rationale of course is that WSDL descriptions can be referenced by our service directory (UDDI), and the combination of WSDL and UDDI is expected to promote the use of Web services worldwide. When we made this decision version 2.0 was in draft status. However, the .NET Framework 1.1 generates WSDL documents compatible with WSDL 1.1 and we plan on using the generated documents as starting points for our WSDL contracts.

Of course, WSDL 1.1 defines only a SOAP/HTTP binding. In our architecture we also want to allow services to be invoked over MSMQ for reliability. As a result we needed to publish a WSDL extension schema we could use to add information that service consumers could use to invoke our services using MSMQ and receive responses.


<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace=
"http://schemas.compassion.org/common/wsdl/2005-04-14" elementFormDefault="qualified" attributeFormDefault="unqualified" id="CIWSDL">
<xs:element name="requestQueue">
<xs:complexType>
<xs:attribute name="location" type="xs:anyURI" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="responseQueue">
<xs:complexType>
<xs:attribute name="location" type="xs:anyURI" use="required"/>
</xs:complexType>
</xs:element>
</xs:schema>

As you can see this schema includes requestQueue and responseQueue elements that are to be used in the WSDL 1.1 port element to identify the MSMQ queues through which services can be invoked and responses received.

The following is an example WSDL document that uses the WSDL extension schema.


<?xml version="1.0" encoding="utf-8"?>
<wsdl:definitions
xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns:s="http://www.w3.org/2001/XMLSchema"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:tns="http://tempuri.org/"
xmlns:tm="http://microsoft.com/wsdl/mime/textMatching/"
xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/"
targetNamespace="http://tempuri.org/"
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:ca="http://schemas.compassion.com/common/wsdl/2005-04-14/">
<wsdl:types>
<s:schema elementFormDefault="qualified"
targetNamespace="http://tempuri.org/">
<s:element name="HelloWorld">
<s:complexType />
</s:element>
<s:element name="HelloWorldResponse">
<s:complexType>
<s:sequence>
<s:element minOccurs="0" maxOccurs="1"
name="HelloWorldResult" type="s:string" />
</s:sequence>
</s:complexType>
</s:element>
</s:schema>
</wsdl:types>
<wsdl:message name="HelloWorldSoapIn">
<wsdl:part name="parameters"
element="tns:HelloWorld" />
</wsdl:message>
<wsdl:message name="HelloWorldSoapOut">
<wsdl:part name="parameters"
element="tns:HelloWorldResponse" />
</wsdl:message>
<wsdl:portType name="Service1Soap">
<wsdl:operation name="HelloWorld">
<wsdl:input message="tns:HelloWorldSoapIn" />
<wsdl:output message="tns:HelloWorldSoapOut" />
</wsdl:operation>
</wsdl:portType>
<wsdl:binding name="Service1Soap"
type="tns:Service1Soap">
<soap:binding
transport=http://schemas.xmlsoap.org/soap/http
style="document" />
<wsdl:operation name="HelloWorld">
<soap:operation
soapAction=http://tempuri.org/HelloWorld
style="document" />
<wsdl:input>
<soap:body use="literal" />
</wsdl:input>
<wsdl:output>
<soap:body use="literal" />
</wsdl:output>
</wsdl:operation>
</wsdl:binding>
<wsdl:binding name="Service1SoapMSMQ"
type="tns:Service1SoapMSMQ">
<soap:binding
transport=http://schemas.xmlsoap.org/soap/msmq
style="document" />
<wsdl:operation name="HelloWorld">
<soap:operation
soapAction="http://tempuri.org/HelloWorld"
style="document" />
<wsdl:input>
<soap:body use="literal" />
</wsdl:input>
<wsdl:output>
<soap:body use="literal" />
</wsdl:output>
</wsdl:operation>
</wsdl:binding>
<wsdl:service name="Service1">
<wsdl:port name="Service1Soap"
binding="tns:Service1Soap">
<soap:address
location="http://win2k3dev/ProcessCreditCard/Service1.asmx" />
</wsdl:port>
<wsdl:port name="Service1SoapMSMQ"
binding="tns:Service1SoapMSMQ">
<ci:requestQueue
location="urn:DIRECT=OS:compassion.com\RequestQ" />
<ci:responseQueue
location="urn:DIRECT=OS:compassion.com\ResponseQ" />
</wsdl:port>
</wsdl:service>
</wsdl:definitions>

Here the WSDL contract defines two ports. The first is a SOAP port that uses HTTP as the transport while the second is a SOAP port that uses MSMQ as the transport. The requestQueue and responseQueue elements are located inside the port.

It should be noted that the transport is defined in the binding element located within the Service1SoapMSMQ element. By convention, and utilizing section 3.3 of the SOAP 1.1 specification (http://www.w3.org/TR/wsdl), the transport attribute is set to http://schemas.xmlsoap.org/soap/msmq to denote to a service consumer that the binding uses MSMQ as a transport.

Service Discovery

2005-05-16T19:37:00.000-07:00

The first of the seven categories in our "Services Platform" is Service Discovery. We defined this category as "The automatic identifying of a software-based service which allows processing functions to be offered and then executed after they have been located. Also includes design time notification". In determining what our standard would be we developed a set of requirements and use cases to describe the desired functionality.

The requirements are:

The ability to publish services in a searchable repository (service directory).
The ability for a developer to view available service bindings for a service within the IDE and select one.
The ability for a software developer to see available service bindings for a service and select one.
The ability for a repository to maintain a list of subscribers.
The ability for a repository to send email notification to subscribers.
The ability for a repository to detect a breaking change to a service contract.
The ability for a developer to register a service as a subscriber to a particular service provider for the purposes of impact analysis.
The ability for a service directory to have high availability by running on multiple nodes that are network load balanced.

And the use cases are:

Publish a Service. At design time, the developer publishes the new service to the service repository via a web-based user interface.
Modify a Service. At design time, the developer modifies an existing service and publishes the modified service via a web-based user interface. If this is a breaking change, then this invokes the notification use case below.
Search for a Service. At design time, the developer needs the ability to query a service repository to discover whether or not a specific service exists to determine whether to build a new one or employ existing one (not re-inventing the wheel).
Select a Service Endpoint. At design time, the developer needs to choose among service endpoints (e.g., MSMQ, .NET Remoting, SOAP) based on the relevant system requirements
Service Directory Notifies Subscribers of Contract Change. At design time, the service directory sends an email notification to a subscribing set of developers when an existing service contract changes. This is done for the sake of impact analysis.
Developer Subscribes to a Service. At design time, the developer registers their application as a consumer of a particular service that is listed in the service directory. Registration involves providing the mailing list (Exchange Group, but preferably not an individual email address) of the developer responsible for responding to contract changes. The value that this provides is the ability for the notificaiton use case to know who the service consumers of a particular service are and how to notify them when a breaking change occurs
Developer Un-Subscribes to a Service. At design time, the developer un-registers their application as a consumer of a particular service that is listed in the service directory. This terminates the ability for notification.
Application Discovers Service at Startup. At runtime during application startup, the application needs to discover initially/dynamically what channel/binding resolves to a specific endpoint and what the service contract is at that endpoint (provided by service directory). Note: It is a best practice to persist (cache) in the application the service data from most recent contact with the service directory. This mitigates any issues should the Service Directory be unavailable. This will likely be implemented in a template class that is used to derive classes that need this functionality.
Application Discovers Service Change at Runtime. At runtime during application execution, the application needs to be able to discover dynamically if an endpoint changes or whether a contract changes and take appropriate remedy action.

As we looked at this list we found that Universal Description, Discovery and Integration (UDDI) 2.0 as implemented on Windows Server 2003 satisfied some but not all of these requirements. Where UDDI is weak (or rather non-existant) is on the subscription and notification end.

After discussing the situation with Microsoft we decided that we would provisionally adopt UDDI 2.0 on Windows Server 2003 as the standard for Service Discovery and then later (after our first deliverable) add our own extensions to provide the subscription and notification services we'll need in the long term. We felt we could afford to follow this course since most clients in our architecture will interact with the directory through a Service Agent or "proxy" that is client-side. In fact, as I'll describe later, our Service Agent abstracts the calls to the directory at runtime (the last two use cases) using the interface defined below so that we can later switch directories or implement our own.
namespace Compassion.Services.Common.Directory { public interface IServiceDirectory { IServiceEndpoint RetrieveServiceEndpointUsingUuid (string type, string serviceKey);IServiceEndpoint RetrieveServiceEndpointUsingServiceInterface(string type, string serviceInterface); void SetConnection(string url, bool secure); } }
We've then implemented a UddiServiceDirectory class that inherits this interface and uses the Microsoft UDDI SDK to interact with the directory. You'll notice that the basics of the interface allow for retrieving the service endpoint (an object that encapsulates the communication transport) using either the service key (a GUID like that used in UDDI) or the name of the service interface. The type parameter is used to determine which service endpoint will be returned. In our first go round we've defined "Default" and "Reliable" that map to an HTTP or MSMQ end point respectively.

And even though the UDDI 3.0 specification is published, only UDDI 2.0 is implemented on Windows Server 2003 and so that gives us the core level of functionality (the first three requirements) out of the box that we don't need to implement.

Service Taxonomy

2005-05-13T05:40:00.000-07:00

As promised I will get back to the services platform we're following here at Compassion. But before doing so I wanted to discuss briefly our concept of "service taxonomy". Basically, we've defined four types of services that have different characteristics. The idea is that these different types of services may be implemented in different technologies that are suited to those characteristics. They include:

Process Services

Exposes a business process
Typically long running and invoked using a one-way or notification (reservation) pattern
Defines exceptions, escalations, and/or retries
Encapsulates rules particular to the business process
Invokes entity services to persist data and infrastructure services to perform steps in the business process
Includes transaction control using compensation
Includes rules particular to the business process
One process service may act as a controller for another and invoke it

Activity Services

Exposes business activities that encapsulate calls to multiple entity services and perhaps infrastructure services to perform a single activity
Each operation scoped as a logical unit of work
Typically immediate request-response pattern
Includes rules that are common to the organization
Returns exceptions

Entity Services

Exposes business entity and associated reference data
Typically immediate request-response pattern
Uses a published schema to represent the entity
Returns exceptions
Encapsulates data consistency rules and CRUD+ that are common to the entire organization
May invoke infrastructure services for data validation
Each operation scoped as a logical unit of work

Infrastructure Services

Exposes shared functionality that is not specific to a business process or entity (not domain specific)
Typically immediate request-response
May abstract interactions with third party services or legacy systems
Returns exceptions
Does not invoke other services

More about how we have decided to implement each of these soon.

Tenets of Service Orientation

2005-05-04T12:24:00.000-07:00

Before I get into the service standards it's appropriate to lay the groundwork by defining what we mean by service-orientation or service oriented architecture. Much of this is probably familiar to many readers and is based the lead of some key Microsoft architects, but is provided as a stake in the ground.

Although definitions of this term vary in the industry the idea within Compassion an SOA or service-oriented architecture is an orchestrated sequence of messaging, transformation, routing and processing events in which XML technologies expose the message content and the components that operate on the messages.

Technically a service exposes an interface contract that encapsulates operations within a specific domain (business or technical for example). This contract defines the behavior (policy) of the service, the messages used to interact with the service, and the message exchange patterns the service employs. The service manages its own state information. Because the interface contract is platform and language independent, the technology used to define messages must also be agnostic with respect to any specific platform/language. Therefore, messages are constructed using XML documents that conform to XML schemas (XSD). Services then exchange messages with applications or other services.

In short service-orientation encompasses the following tenets:

Explicit Boundaries – services are bounded by their contracts that define specific messages they accept and return
Autonomous – services are self-contained and built to last. The applications that use them will change more frequently than the services themselves
Negotiation via policy – services expose their requirements such as security and message exchange patterns via policies in the contract
Exposed Schema and Contract – service consumers use the contract and the schema it contains in order to create messages that interact with the service
Message Driven – all interaction with the service occurs through messages. Messages share schema and contract but not class or internal representation
Reliable - services are by definition shared resources and so is not the purview of a single application. In other words services are built to last, applications and business processes are built to change. As a result there is an implicit requirement that services be reliable.
Available – just as in the previous tenet services must be highly available since they will be shared by a wide variety of applications

Defining a Services Platform

2005-05-04T11:48:00.000-07:00

In my previous post I discussed the steps Compassion was taking in order to implement SOA. As promised I won't bore you with business process specifics (step number 1) but it might be of interest to many of you the SOA standards we've adopted. These were written in large part by a extremely talented fellow Compassion architect named Chuck Boudreau (and owner of Golden Hills Software and the SurveyGold product).

The way we approached this task is to define a "services platform" that includes seven key categories in which we published standards for our IT organization. These categories describe essential services necessary to enable effective communications between a service consumer and a service provider. The idea of doing so was based on advice from the book Understanding SOA with Web Services by Newcomer and Lomow.

Here they are:

Service Discovery
The automatic identifying of a software-based service which allows processing functions to be offered and then executed after they have been located. Also includes design time notification

Service Contract
The request and (optional) response schemas for each operation exposed by a service, the security protocols, and the bindings associated with a service automatic identifying of a software-based service.

Service Agents (Proxies)
These isolate the idiosyncrasies of calling diverse services from your application, and can make available additional services, such as basic mapping between the format of the data exposed by the service and the format your application requires.

Communication
The envelope schema including addressing and context to support routing and correlation of messages sent between service consumers and service providers.

Security
Message encryption, service consumer authentication and security consumer authorization and how authorization is managed.

Message Exchange Patterns
Service contracts express valid message exchange patterns between two parties. The party that initiates communication is called the initiator. The other party is called the service. Service contracts specify one or more operation contracts. An operation contract represents an individual message exchange or a correlated request/reply message exchange

Management/Quality
Addresses operational and management concerns including logging, performance monitoring, duplicate message handling and exception management.

In the posts that follow I'll drill down into each of these and describe what is included in each standard and a little about how we arrvied at the standard.

SOA Step by Step

2005-04-27T23:04:00.000-07:00

At Compassion the decision was made to move towards service orientation in order to support an ever growing number of sponsored children. Currently, Compassion manages data on over 650,000 children who are sponsored by individuals and groups around the world. That number is likely to grow to over one million children by sometime in 2007. By constructing new systems using the principles of service orientation and introducing business process management software, the idea is to build systems composed of processes and services that can be scaled out to meet that demand.

The process of moving from client/server development (a monolithic Powerbuilder application performs almost all business functions today with a SQL Server backend) to service orientation is a challenging one to say the least. The general path that we’ve chosen is the following:

Identify a project (a set of business processes) where service oriented architecture can be implemented along side existing systems
Evaluate the SOA landscape and publish a set of SOA standards within the IT organization
Build a reference implementation or working model that uses the standards
Use the reference implementation as the template for implementing the project

Compassion is a Microsoft shop and so the tools we’ll be using in this endeavor include Visual Studio, BizTalk, SQL Server, IIS, and MSMQ as well as a business process software product from Ultimus.

SOA

2005-04-23T19:06:00.000-07:00

Recently my family and I moved from Kansas City to start a new job with Compassion International. As a part of this job I'm working with a great team to implement the standards, architecture, and first project in Compassion's Service Oriented Architecture (SOA) using Microsoft technology.

On this blog I intend to share our experiences as we make our way down this road. Stay tuned.