Brownfield (software Development) - Addressing Environmental Complexity

Addressing Environmental Complexity

Reliably reengineering existing business and IT environments into modern competitive, integrated architectures is non-trivial. The complexity of business and IT environments has been accumulating almost unchecked for forty years making changes ever more expensive. This is because:

• Environmental complexity is often expressed in legacy code. Legacy skills shortages are driving up maintenance and integration costs.
• Existing complex environments must be re-engineered in phases that make operational sense to their associated business function. These phases often default to wholesale, risky replacements of systems as ignorance of existing complexity means that potential incremental changes are too difficult to understand and engineer.
• Accelerated development methods have left enterprises with modern legacy systems. Complex Java and .NET applications have many of the same problems as older COBOL applications.

As a result, an increasing proportion of the effort of developing new business capabilities is spent on understanding and integrating with the existing complex system and business landscape rather than delivering value. It has been observed that up to 75% of overall project effort is now spent on software integration and migration rather than new function.

The IT industry as a whole has a poor success rate at delivering such large scale change for its clients. The CHAOS survey from the Standish Group has tracked an overall improvement in IT project delivery success over the last twenty years, but even in 2006 large IT projects still failed more often than succeeded. Engineering changes and in such environments has many parallels with the concerns of the construction industry in redeveloping industrial or contaminated sites. They are full of hazards, unexpected complexities and tend to be risky and expensive to redevelop. The accumulated complexity of IT environments has made them “Brownfield” sites.

Ironically it is not the complexity of the new function or any new system characteristics that are the root of large project failures – it is our understanding and communication of the overall requirement (as identified in The Mythical Man Month). To succeed the requirements need to include a precise and thorough understanding of the constraints of the existing business and IT. Current “Greenfield” tooling and methods use early, informal and often imprecise abstractions that essentially ignore such complexity. However the devil is always in the detail. Early, poorly informed abstractions are usually wrong and are often detected late in construction, resulting in delays, expensive rework and even failed developments. A Brownfield oriented approach embraces existing complexity, and is used to reliably accelerate the overall solution engineering process, including enabling phased, incremental change wherever possible.

Brownfield takes the standard OMG model/pattern driven approach and turns it on its head. Rather than taking the conventional approach of starting with a Conceptual Model and driving down to Platform Specific Models and code generation, Brownfield starts by harvesting code and other existing artifacts and uses patterns to formally abstract upwards towards the Architecture and Business tier.

Standard Greenfield techniques are then used in combination to define the preferred business target. This “meet in the middle” technique is familiar from other development methods, but the extensive use of formal abstraction and the use of patterns for both discovery and generation is novel.

The underlying conceptual architecture of all Brownfield tools is known as VITA. VITA stands for Views, Inventory, Transformation and Artifacts. In a VITA architecture, the problem definition of the target space can be maintained as separate (though related) native "headfulls" of knowledge known as Views. The core advantage of a View is that it can be based on pretty much any formal tool. Brownfield does not impose a single tool or language on a problem space – a core tenet is that the headfulls continue to be maintained in their native forms and tools.

Native Views are then brought together and linked into a single Inventory. The Inventory is then used with a series of Transformation capabilities to produce the Artifacts that the solution needs.

Views can currently be imported from a wide variety of sources including UML, XML sources, DDL, spreadsheets etc. The Analysis and Renovation Catalyst tool from IBM has taken this capability even further via the use of formal grammars and Abstract Syntax Trees to enable almost any program to be parsed and tokenized into a View for inclusion into the Inventory.

The rapid cyclic nature of the discovery, re-engineer, generate and test cycle used in this approach means that solutions can be refined iteratively in terms of their logical and physical definitions as more of the constraints become known and the solution architecture is refined.

Iterative Brownfield development can allow the gradual refinement of logical and physical architectures and incremental testing for the whole approach, resulting in development acceleration, improved solution quality and cheaper defect removal. Brownfield can also be used to generate solution documentation, ensuring it is always up to date and consistent across different viewpoints.

The Inventory that is created through Brownfield processed may be highly complex, being an interconnected multi-dimensional semantic network. The level of knowledge in the Inventory can be very fine grained, highly detailed and interrelated. Such things are hard to understand and can provide barriers to communication however. Brownfield solves this problem by abstracting concepts via an artisan’s best guess, using known patterns in its Inventories to extract and infer higher level relationships.

Formal abstractions enable the complexity of the Inventory to be translated into simpler, but inherently accurate, representations for easier consumption by those that need to understand the problem space. These abstracted Inventory models can be used to automatically render multi-layered architecture representations in tools such as Second Life.

Such visualizations enable complex information to be shared and experienced by multiple individuals from around the globe in real time. This enhances both understanding and a sense of a single team.