Always Learning

Last week we rolled out the last release in this year long project that developed and then proved out the service oriented infrastructure. Although we have issues to deal with related to the proper implementation of some of the uses cases (more on that in a later post) I thought I'd take a few minutes to discuss what I think is perhaps one of our biggest lessons learned.

In designing the contracts for our services we purposely wanted to introduce a layer of abstraction into the contracts so as to avoid creating service interfaces that were too closely tied to the existing enterprise data structure. Of course that's a balancing act since on the one had you need to have enough information represented in the various entity schemas to accurately identify the entity, and on the other hand not include extraneous attributes that serve to clutter the picture.

What we did well in this process is create the larger logical model where we introduced three high level concepts - Constituents, Needs, and Relationships. The first represents individuals of interest to Compassion, the second are entities whose needs can be satisfied such as a child to sponsor, a fund to donate to, or even an activity like praying for the work of the ministry. Relationships then represent the various connections between Constituents and Needs. For example, a Prayer Relationship is established between a Constituent and a Prayer Need while a Payment Relationship is established between a Constituent and a Child, Project, or Fund Need. This structure served us well because it allowed us to subdivide and represent each action for a constituent with a particular relationship type and more importantly allow our messaging infrastructure in BizTalk to recognize these relationship types and create and farm out work to independent orchestrations through a publish and subscribe model. This is an extensible model so that as new relationships are defined the infrastructure can handle them in a loosely coupled fashion (in fact, we implemented a new kind of action in this release without any modifications at all to the core BizTalk messaging model). So far so good.

The problem entered because going in we (and by that I mean I) didn't really understand how the logical model within the Payment Relationship should be defined. In other words, the Payment Relationship being the most complex, it needed a more robust definition at the outset in order to handle all of its variance. Unfortunately we rolled out the first definition of the Payment Relationship schema in our previous release when we were dealing only with relatively simple relationships that did not involve funded donations and commitments (in other words, we were dealing only with promises to pay and not persisting and handling rules for credit card and ACH transactions). And while there were place holders in the schema for this kind of information in the first version, the basic structure didn't extend as well as expected to handle this extended information. As a result, and here is the big idea, we ended up adding attributes to the schema that were too closely tied to the existing relational data model. To put it simply, too many codes right out of the database and not enough abstraction between the messaging model and the persistence layer. What happens when you do this of course is that you not only bring along the codes but their interrelationships and dependencies as well. Had we done more analysis up front, we would have seen these relationships and probably created higher level abstractions to handle them within our service contracts.

In one sense I'm glad we could roll out such a sophisticated system in a series of four iterations over a year, but in another the architectural tradeoffs that resulted from this approach (since we did not have the resources nor the time to fully flush this out ahead of time) will likely hamper us in ways as move down the road. But then again you're always living and learning and so we've toyed with the idea of version 2 service operations in some cases that provide these higher level abstractions.

Transitioning

For our project we're now about to release our final deliverable. This is the fourth deliverable in the project with the first one being in August of last year. This one is the largest and is used to automate many of the financial transactions performed on our web site which will in turn be used in the automation of requests that come from other channels (scanned forms and external call centers).

That said, the frameworks that the team has built are about to be used by other project teams as well. To that end one of the talks I attended at TechEd was a discussion of on assisting other developers in using framework that you build. One of the points made was that when you’re developing frameworks (like we have for services, orchestrations, and human workflow) the level of effort for framework developers doesn’t necessarily go down over time if you continue to refine it. The illustration used was the following graph.



As a result, the next effort for the team will be to transition from a project team doing business development into a team that continues to refactor and improve the frameworks that other developers use.

One of the other points made, however, was that teams should invest the time in the framework up front in order to make it consumable through the use of code generation, templates, etc. The presenter specifically mentioned that trying to put something together in a just in time fashion when other teams need it doesn’t work well in his experience. Unfortunately, because of the very tight schedule we've been on the past year that is largely where we are. The key insight for other teams doing this kind of work is to budget the time (I know how difficult that is when the work is viewed as "overhead" of course) in each release to refactor your frameworks and (reference architecture) and incorporate guidance as you go as well as allow time for mentoring other teams.