The un-impressive track record in the development of BI applications led us to conceive an improved approach to reduce the initial investment required to launch a BI project in order to deliver key performance indicators (KPIs) much more quickly and at lower cost.

The incremental development approach we came up with is aligned with the business priorities while allowing business users to further develop the knowledge of their business and better define their expectations.

As opposed to the traditional waterfall approach to building a BI application which is: plan the project, define the requirements, prepare the architecture, develop, test, move to production in a sequential manner, we propose to use an iterative approach during which we build layers of the entire data warehouse through iterations.

For example, instead of spending time in defining in a high level requirements, work with the end users to understand their business and the indicators they use to successfully manage it. Using an iterative approach, the objective of our process is to work backward starting with the required performance indicators back to the source systems.

As can be seen in the diagram, the intent is not to divide the project into phases where data is extracted from the source systems first, then move to a centralized repository and so forth. Instead of the traditional approach, we use a layer approach where each indicator is developed from source system to the presentation layer.

Although our approach requires a substantial paradigm shift for most people having experienced a traditional waterfall approach for BI
projects, the notions we propose are simple, logical and straightforward. This approach relies on AGILE principles and
more specifically on SCRUM practices that have been demonstrated to be successful in other types of initiatives. Our objective is to present them in a very concise manner in this post.

Our proposed approach to BI development can be summarized by the following principles:

• Segment the development of components into small iterations of 3 to 4 weeks and frequently deliver key performance indicators at the end of each iterations;
• Repeat the iterative process and use increments to add components to the BI architecture and infrastructure;
• Prioritize the development of the key performance indicators that will deliver the highest value to the organization first;
• Using a top-down approach going from aggregation to granularity, develop each KPI from end-to-end or from source systems to presentation layer;
• Avoid “analysis-paralysis” forces by using a “good enough” approach in documentation, architecture, modeling, project management
  and other overhead activities;
• Learn from each of the iteration and adapt the process appropriately to increase efficiency.

Our approach promotes a process that encourages frequent inspection and adaptation, a set of engineering best practices that allow for
rapid delivery of high-quality software, and a business approach that aligns development with customer needs and company goals. With this
approach, the project delivers in small increments with minimal planning, rather than long-term planning.