My Rating: 9 / 10
I read Agile Data Warehousing: Delivering World-Class Business Intelligence Systems Using Scrum and XP
a few weeks ago and I thought the author did an amazing job at demonstrating how the Agile approach can be used for Data Warehousing projects. A such, I wanted to spend some time providing a more detailed summary of this book.
Agile Data Warehousing (ADW) is the application of two agile development approaches SCRUM and XP (extreme programming) – to the specific challenges of data warehousing and business intelligence.
The new approach is being proposed to address lamentable success rates of waterfall BI projects and as such ADW addresses the following problems:
- Waterfall software development historically yielded notoriously poor ROI;
- Customers are increasingly impatient to get BI benefits;
- BI cannot avoid tackling the unknown;
- Infrastructure problems cause problems late in the project;
- Business rules can also appear late in the plan due to changes in the environment;
- Project often take fatal risk in order to avoid a little rework;
- Requirements definition and review cycles consume excessive efforts;
- Documentation has a short shelf life;
- Standard project-management consumes large portions of the project budget;
- Complex methods and elaborate plan rarely perform well;
- BI need more time to acclimate the project;
- Customers need more time to learn about BI.
If good decisions allow a company to market its products or services more accurately, reduce its production costs or amplify its ability to fill orders, then a software engineering method leading to faster and better development of data warehouse offers much more than cost savings. The proposed approach is based on the following 4 Agile principles:
- Individuals and interactions over processes and tools;
- Working software over comprehensive documentation;
- Customer collaboration over contract negotiation;
- Responding to change over following a plan.
The principles behind the agile manifesto are known but they are repeated to ensure a common understanding.
- Our highest priority is satisfied customer through early and continuous delivery of valuable software.
- Welcome changing requirements even late in development agile processes are changed for the customers competitive advantage.
- Deliver working software frequently from a couple of weeks to a couple of months with a preference for the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals, defend the environment and supporting to entrust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors developers and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity is the art of maximizing the amount of work done is essential.
- The best architectures requirements and designs emerge from self organizing teams.
- At regular intervals the team reflects on how to become more effective than tunes and adjusts its behavior accordingly.
- Even the best approach needs to be cultivated and refined into a step-by-step method that teams can follow.
Naturally, achieving breakthrough performance in the BI development requires some significant changes:
- The physical environment developers work in.
- The role of the project manager.
- The way we do requirements.
- The way we do estimates.
- The way we do design.
- The way we code and tests.
- The way we document.
- The way we implement the system.
- The way we relate to other corporate departments such as information systems.
- The way we relate to other business stakeholders besides the project sponsor.
The five stages to migrate to an Agile approach are summarized below.
Stage 0: Co-located self organized team
- Co-locating the team members in a shared workspace surrounded by dry erase board, where they can work closely together, relying on light weight, verbal and diagrammatic communication for collaborative design and coding.
- Embedding the customer into the team as a product owner so that she can work closely with the developers to build the application the organization needs.
- Acting team members to organize their work in any way they prefer as long as they can transform a significant portion of the product owner’s request into potentially shippable code every 2 to 4 weeks.
- The team works in the minimalist environment for a few iterations, with the quality of his work review and approved by the product owner at the end of each iteration of development. The work product during this preliminary phase can be incomplete and lack polish.
- The customer does become fully involved and the team finds ways to meet pressing the deadline without wasting effort on that to be documentation.
Stage 1 – User epic decomposition and estimation skills
- This stage introduces the teams guiding product owner to several BI specific architectural notions so that the team begin acquiring a common language and so that many intertwined details of the full business intelligence implementation get addressed in an orderly fashion.
- The developers learn to estimate their work using the rim of the relative size of requirement. That technique which they will use to scope each development iteration.
Stage 2 – Release planning
- The team tests the full circle of the desired software release and the number of iteration it will take for it to deliver the remainder of the data warehouse project.
Stage 3 – Reference model and test-led development
Stage 4 – Pipeline deliveries squads
Stage 5 – Continuous integration testing
- By establishing an integration environment into which every module is integrated. As soon as a developer declared it done.
During the project, the team follow an iterative process to deliver incremental business value.
Planning phase
During the planning phase stakeholders represented in the team work through a list of business needs that might be addressed through new software development and agree upon which of them will be within the scope of the next release of the application.
Requirements gathering
Requirements are called users stories and together they make the release backlog.
Build phase
The build phase of the release cycle involves many iteration or even shorter cycle the multi week development sprint in which the real coding work of the project will be done. The modules produced during each sprint are added to the deliverable of prior sprints until the team feels the entire collection represent an acceptable release which is then deployed.
The build phase of the release cycle consists of multiple development iteration called sprints. The sprints are likely structured time during which the team design and code the software modules needed.
Retrospective
The release cycle concludes with a review process where lessons learned can be identified and used to improve the effectiveness of the next turn of the cycle.
During this retrospective the team quantifies its velocity – the amount of work is successfully delivered within the time off of the sprint just completed. After a few sprints, the team velocity stabilizes into a predictable range allowing project folders to group the story in the release backlog using sprint-sized bracket and thereby forecasts how many time boxes it will take the delivered the entire release.
