FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

Resist the Urge to Start Coding

Kimball Perspectives

DM Review welcomes Ralph Kimball as a new columnist. The Kimball Perspectives series will systematically describe classic best practices as well as new trends in technologies.

The most important first step in designing a data warehouse (DW)/business intelligence (BI) system, paradoxically, is to stop. Step back for a week, and be absolutely sure you have a sufficiently broad perspective on all the requirements that surround your project. The DW/BI design task is a daunting intellectual challenge, and it is not easy to step far enough back from the problem to protect yourself from embarrassing or career-threatening changes “discovered” after the project is underway.

Before cutting any code, designing any tables or making a major hardware or software purchase, take a week to write down thoughtful, high-quality answers to the following 10 questions, each of which is a reality that will come to control your project at some point. These define the classic set of simultaneous constraints faced by every DW/BI effort.

Classic Questions to Ask

  1. Business requirements. Are you in touch with the key performance indicators (KPIs) your end users actually need to make the decisions currently important to your enterprise? Although all 10 questions are important, understanding the business requirements is the most fundamental and far reaching. If you have a good answer to this question, then you can identify the data assets needed to support decision-making, and you will be able to decide which measurement process to tackle first.
  2. Strategic data profiling. Have you verified that your available data assets are capable of supporting the answers to question number one? The goal of strategic data profiling is to make go or no-go decisions very early in the DW project whether to proceed with a subject area.
  3. Tactical data profiling. Is there a clear executive mandate to support the necessary business process re-engineering required for an effective data quality culture, perhaps even driving for Six Sigma data quality? The only real way to improve data quality is to go back to the source and figure out why better data isn’t being entered. Data entry clerks are not the cause of poor data quality! Rather, the fixes require an end-to-end awareness of the need for better quality data and a commitment from the highest levels to change how business processes work.
  4. Integration. Is there a clear executive mandate in your organization to define common descriptors and measures across all your customer-facing processes? All of the organizations within your enterprise who participate in data integration must come to agreement on key descriptors and measures. Have your executives made it clear that this must happen?
  5. Latency. Do you have a realistic set of requirements from end users for how quickly data must be published by the data warehouse, including as-of-yesterday, many-times-per-day and truly instantaneous?
  6. Compliance. Have you received clear guidance from senior management as to which data is compliance sensitive, and where you must guarantee that you have protected the chain of custody?
  7. Security. Do you know how you are going to protect confidential as well as proprietary data in the ETL back room, at the end-users’ desktops, over the Web and on all permanent media?
  8. Archiving. Do you have a realistic plan for very long-term archiving of important data, and do you know what data should be archived?
  9. Supporting end users. Have you profiled all your end-user communities to determine their abilities to use spreadsheets, construct database requests in ad hoc query tools or just view reports on their screens?
  10. IT licenses and skill sets. Are you prepared to rely on the major technology site licenses your organization has already committed to, and do you have enough staff with advanced skills to exploit the technical choices you make?

Time spent answering these classic DW questions is enormously valuable. Every one of the answers will affect the architecture, choice of approaches and even the feasibility of your DW/BI project. You dare not start coding before all the team members understand what these answers mean!

The big news is that end users have seized control of the DW. They may not be building the technical infrastructure, but they are quite sure that they own the data warehouse and the BI tools and that they must meet their needs. This transfer of initiative from IT to the end users has been very obvious in the last two or three years. Witness the soul-searching articles and industry speeches exhorting CIOs to show more business leadership and the high CIO turnover as reported in CIO Magazine.

Many of the 10 questions in this column are brought into much clearer focus by increased end-user ownership of the DW/BI system. Let’s focus on the top five new urgent topics, in some cases coalescing our questions:

Business requirements.The DW/BI system needs a permanent “KPI team” continuously in touch with end-users’ analytic needs and the consequent demand for new data sources to support new KPIs. Also, the system should increasingly support the full gamut of analytic applications, which include not only data delivery, but alerting the users to problems and opportunities, exploring causal factors with additional data sources, testing what-if scenarios to evaluate possible decisions and tracking the decisions made. The DW/BI system is not just about displaying reports, but rather must be a platform for decision-making in the broadest sense. The oldest label for data warehousing, decision support, remains surprisingly apt.

Strategic data profiling. The earlier you tell the users bad news about the viability of a proposed data source, the more they will appreciate you. Develop the ability to assess a data source within a day or two. Elevate the data profiling tool to a strategic, must-have status.

Tactical data profiling. The increased awareness of data quality is one of the most remarkable new DW perspectives, certainly driven by end users. But all is for naught if the business is not willing to support a quality culture and the end-to-end business process re-engineering required.

Integration and latency. The end-user demand for the 360-degree integrated view of the business has been more like an approaching express train than a shock wave. We have been talking about it for almost a decade. But now the demands of integration coupled with real-time access to information have combined these two issues into a significant new architectural challenge.

Compliance and security. DW/BI folks in IT often don’t have the right instincts for protecting data, since the system is supposed to be about exposing data. But this new emphasis on compliance and security can be built systematically into the data flows and the end-user tools across the entire DW/BI system.

The purpose of this first column has been to expose the fundamental design issues every DW/BI design team faces and to bring to the surface the urgent new requirements. In this ongoing series of columns, I will have the chance to probe each of these areas in some depth, reminding us of the remarkably unchanging aspects of data warehousing, while at the same time trying to catch the winds of change.

Ralph Kimball is the founder of the Kimball Group and Kimball University where he has taught data warehouse design to more than 10,000 students. He is known for the best selling series of data warehouse "Toolkit" books. He started with a Ph.D. in man-machine systems from Stanford in 1973 and has spent the last 34 years designing systems for end users that are simple and fast. You can reach him at ralph@kimballgroup.com.

For more information on related topics, visit the following channels:



Industry Vendors