Recovering from the Accidental Architecture

In last month’s “The Accidental Architecture” column, I discussed how enterprises have evolved their data warehousing (DW) and business intelligence (BI) projects without a planned architecture. This evolution was the result of developing tactical projects while chasing the latest technology proclaimed by analysts, columnists and the vendors offering these solutions.

The random walk that created this accidental architecture may have solved some immediate, tactical needs, but the expenditure of time, resources and budget didn’t deliver ROI. Even worse, each new project resulted in a new stovepipe of data and technology that required increasing resources to integrate. It also made business users more likely to develop data shadow systems to perform their own integration.

How do you break the cycle of dependency on hype and the allure of silver-bullet technologies? In order to correct a problem, you have to recognize you have a problem. Both business and IT need to confront the accidental architecture addiction and resolve to work a better way - without finger-pointing.

Don’t make the mistake of thinking you can use a green field solution to fix your accidental architecture. You do not have a green field, and it is not practical to start from scratch. You have various silos of data and technologies, including data shadow systems. Whatever your solution architecture, you need to plan on building it iteratively and replacing or integrating your silos incrementally. Your solution architecture needs to be built using a program approach; it will evolve as your business and technology evolve.

Take these steps toward developing the architecture:

• Review your documentation of existing systems.
• Reverse-engineer any systems without updated documentation or where there are gaps in the documentation.
• Gather and prioritize business requirements.
• Gather and prioritize technology considerations.
• Design your architecture.
• Determine the gaps between your current systems and your solution architecture.
• Develop a program plan, including budget, resources and timetable to get you toward that architecture.
• Start your first project.

A key consideration is that the architecture has four components: information, data, technology and products. Resist the urge to evaluate and select products. You need to first understand and agree on the information architecture that your business needs. Then determine the data you need, the condition of that data and what you need to do to cleanse, conform and transform that data into business information.

Next, determine what technologies (not products) are required by the information and data architectures. Finally, almost as an afterthought, evaluate and select products. Let me also suggest that you consider the products you may already have in your silos as incumbent candidates and determine if they can take you on your journey. Do not chase the latest and greatest if your incumbent products can get the job done.

Rather than looking at products, you should spend your valuable time examining the following areas to drive what and how your solutions architecture - information, data and technology - should be.

• Reporting and analytics needs: What information (data and business transformations) do you need?
• Source data: Where do you get the data that you need to create information?
• Data quality and consistency: What levels of quality and consistency do you need (versus want)?
• Performance measures and transformations: How do you transform your source data, and what business information metrics do you need?
• Design data workflow and processes to transform data from sources to information consumption, i.e., reporting and analysis, including how you manage and measure data quality and consistency.
• Design data structures to support business: What data architecture do you need to build (i.e., DW, data marts, cubes, aggregations, denormalized files, etc.)?
• Determine technology to support architecture: What technologies are required to build the data architecture?

This is a lot of work! Designing an architecture to support your enterprise reporting and analytical needs is a significant project that will result in correspondingly significant business ROI, if you do it right. Don’t be intimidated or overwhelmed. Too many people will decide that they do not have the time or money for this and will go back to the tactical project approach. That is shortsighted and will result in greater time and money spent on results that are far short of what you should be achieving.

A word of caution: do not get too wrapped up in the architecture. Some companies will get so fixated on the final architecture that they take months or years trying to develop it. The architecture is not the result of your BI/DW project, but rather a means to an end. Do not spend time on a monstrous, complicated architecture that solves world hunger; design something that you can start developing toward and that you can evolve over time.