Always Learning

Last week we rolled out the last release in this year long project that developed and then proved out the service oriented infrastructure. Although we have issues to deal with related to the proper implementation of some of the uses cases (more on that in a later post) I thought I'd take a few minutes to discuss what I think is perhaps one of our biggest lessons learned.

In designing the contracts for our services we purposely wanted to introduce a layer of abstraction into the contracts so as to avoid creating service interfaces that were too closely tied to the existing enterprise data structure. Of course that's a balancing act since on the one had you need to have enough information represented in the various entity schemas to accurately identify the entity, and on the other hand not include extraneous attributes that serve to clutter the picture.

What we did well in this process is create the larger logical model where we introduced three high level concepts - Constituents, Needs, and Relationships. The first represents individuals of interest to Compassion, the second are entities whose needs can be satisfied such as a child to sponsor, a fund to donate to, or even an activity like praying for the work of the ministry. Relationships then represent the various connections between Constituents and Needs. For example, a Prayer Relationship is established between a Constituent and a Prayer Need while a Payment Relationship is established between a Constituent and a Child, Project, or Fund Need. This structure served us well because it allowed us to subdivide and represent each action for a constituent with a particular relationship type and more importantly allow our messaging infrastructure in BizTalk to recognize these relationship types and create and farm out work to independent orchestrations through a publish and subscribe model. This is an extensible model so that as new relationships are defined the infrastructure can handle them in a loosely coupled fashion (in fact, we implemented a new kind of action in this release without any modifications at all to the core BizTalk messaging model). So far so good.

The problem entered because going in we (and by that I mean I) didn't really understand how the logical model within the Payment Relationship should be defined. In other words, the Payment Relationship being the most complex, it needed a more robust definition at the outset in order to handle all of its variance. Unfortunately we rolled out the first definition of the Payment Relationship schema in our previous release when we were dealing only with relatively simple relationships that did not involve funded donations and commitments (in other words, we were dealing only with promises to pay and not persisting and handling rules for credit card and ACH transactions). And while there were place holders in the schema for this kind of information in the first version, the basic structure didn't extend as well as expected to handle this extended information. As a result, and here is the big idea, we ended up adding attributes to the schema that were too closely tied to the existing relational data model. To put it simply, too many codes right out of the database and not enough abstraction between the messaging model and the persistence layer. What happens when you do this of course is that you not only bring along the codes but their interrelationships and dependencies as well. Had we done more analysis up front, we would have seen these relationships and probably created higher level abstractions to handle them within our service contracts.

In one sense I'm glad we could roll out such a sophisticated system in a series of four iterations over a year, but in another the architectural tradeoffs that resulted from this approach (since we did not have the resources nor the time to fully flush this out ahead of time) will likely hamper us in ways as move down the road. But then again you're always living and learning and so we've toyed with the idea of version 2 service operations in some cases that provide these higher level abstractions.