Although I believe in the value of a book on Agile Business Intelligence, I have decided to put this project on hold indefinitely.
The last few months have allowed me to confirm a passion for corporate strategy and as such, I have very little time to move this project forward.
I wish to thank the collaborators who signed up to participate in this project and hope they will have the energy required to continue this initiative.
As part of the collaborative initiative to write an Agile Business Intelligence and Data Warehousing book, I am looking for people to share their best practices. The book will present the tale of a fictitious organization that successfully developed and implemented a Business Intelligence project using Agile principles. If you wish, you may have a look at the preliminary version of the first chapters of the book.
In this section, I would like us to start brainstorming on the best practices for an Agile BI project based on your personal experience. The objective of this discussion is to derive commonalities across the projects, and major differences that made your project specific. Please answer the following questions and add other questions if you feel the need so we can build on our individual perspective.
Since Google Docs doesn’t allow people to edit documents that have been published, send me an email (martin [at] analytical-mind.com) and I’ll share the document so you can start contributing.
Thanks again for joining this collaborative effort.
This blog post presents a preliminary version of the Preface of the collaborative Agile BI book. You can access the various sections currently available and you may register to join this collaborative effort to start contributing. I also invite you to comment this content at the bottom of the blog post.
There was a running gag in an organization we were working with that every Business Intelligence (BI) project required 3 years to complete and would cost over $3 Million. This running gag demonstrated an unfortunate situation – most BI projects are expensive, time and resource consuming and have a low rate of success. Along these lines, very few people would debate that traditional BI projects require heavy investments and result in long delays before providing value to the organization.
Much has already been said about the importance of BI applications in organizations. One of the most obvious and frequently repeated benefits of BI applications is the ability to increase the quality and timeliness of the business decisions being made. This is increasingly true as BI applications are being democratized and are now embedded in the decision making process at every level of the organization and across multiple departments.
In addition, decision makers constantly need an in-depth understanding of their business’ strengths and weaknesses and this is especially true during difficult economic times. The availability of timely and accurate information can help business leaders make the right decisions. Along these lines, the survey of over 1,500 CIOs[1] shows that despite predicted flat IT budget growth in 2009, BI projects remain their number one technology priority.
There isn’t much need to emphasize further the value of early diagnosis and the implementation of timely solutions but in a changing business environment the business needs evolve too quickly for sequential waterfall development methodologies.
Case in point, it is estimated that 60%[2] of BI projects end in abandonment or failure and market data confirms that out of the 3 standard project dimensions – time, resources, and scope – time lines and resources typically exceed the initial plan while scope consistently fall short of the original expectations and requirements. As such, an approach that delivers value early in the project while remaining aligned with the business priorities is recommended.
In addition, estimates show that no more than 20%[3] of business users actually use their BI applications proactively and that a staggering 64%[4] of systems functionalities are rarely or never used. This regrettable situation compelled us to look for an alternate and improved approach to ensure BI projects have greater success rate. More specifically that BI projects deliver the features required by business users (scope) in a timely fashion (time) while controlling expenses (budget).
Our intent is neither to rehash the unfortunate track record of BI projects nor to highlight the most dramatic failures but to propose a different approach to the development of BI projects.
This is a tall order but with the increasing popularity of AGILE iterative and incremental approaches and with repeated success in other specialities; we know using such an approach in the development of BI projects leads to much better outcomes.
The AGILE approach heavily relies on business involvement within the project team and throughout the duration of the BI project. Having direct contact with the business team, the development team will more easily understand the expectations and will increase its knowledge of the business which will make a more knowledgeable and productive team altogether.
In order to achieve our objectives to: reduce the initial investment, accelerate the availability of the information, deliver the right solution to the business users and constantly deliver value to the organization we explained that an AGILE approach is the right solution. Through iterations and increments, we use a “slicing approach” to develop from source systems to presentation layer in order to deliver on the business users’ requirements.
The benefits of an AGILE approach to a BI project are substantial. From our perspective, the most important one for the organization is that it forces the project team to focus on ROI and deliver high value by spending efforts on activities that will bring the highest value.
By reducing the duration of the development cycles the business and development team can modify their release strategy and focus on new priorities that may arise as a consequence of a changing business environment.
[1] Business Intelligence Summit by Gartner, 2008
[2] Standish Group Study Reported at XP2002 by Jim Johnson
[3] Within the context of this white paper and for simplicity, we combine Data Warehouse (DW) projects with Business Intelligence (BI) projects.
It was 2:15 pm on the Friday afternoon and Jordan and the development team were gathered in the meeting room making sure everything was in place for the presentation. Everyone was fairly relaxed with a sense of accomplishment. A few of the developers were making jokes in preparation for the demonstration that was about to take place.
Jordan had facilitated many demonstrations in the last 18 months and he was always impressed to see how pleased the developers were to present their work to their client. The product demonstration gave the developer the opportunity to present their work and get a sense of accomplishment. Over the years, Jordan had been happy to see that most teams he had worked with gelled quickly and their commitment to getting the job done was greatly increased when the team self-organized, as was the case again for this project.
Catherine walked in with a few of her employees. Immediately people from the marketing team initiated discussion with the software development team. Catherine looked at Jordan and said, “Do you remember when our teams sat silently at opposite end of the table waiting for the first person the open fire toward the other group?”
Jordan burst laughing, “Was it that bad? You make it sound as if Marketing and IT were enemies!”
“Weren’t we?” asked Catherine half-jokingly. “When you took over this project, things had slightly improved but with the previous project manager, this project was a disaster”.
“I know and they trapped me into this project. I wonder what I did to deserve this treatment?” said Jordan smiling. “All I can say is, the team and I are really enjoying working on the project. These end of sprints demo are a great way to get together and look at the great work that has been done to date”.
“I agree.”, said Catherine. “Are we going for beers after the presentation?”
Before Jordan could answer, Scott walked in with Alessia.
“We have important people joining us today” joked Catherine.
“That’s because I keep hearing good things about your project” replied Alessia.
“I’d like to ask everyone to take a seat” said Jordan. “We are just about ready to begin”.
“Kim, did you bring the cards so the team can demonstration he features that have been implemented” asked Catherine.
“Yes. Yes, I did” answered Kim. “I’ll sit next to Brian so he can demo them for everyone”.
The room was configured like other training room with a desk in the front of the room where Brian was sitting. His mouse and keyboard were linked to the main monitor that was used to project against the wall. next to Brian was a chair where Kim could sit and read the index cards on which were written the product features. Everyone else took a seat facing in the direction of the projector. Alessia and Scott took two seats available in the first row.
“OK, it’s 2:30 pm. Let’s begin” said Jordan.
“As a customer, I want to be able to add an item to my shopping so I can initiate the checkout process”, said Kim.
With a few click of the mouse and the relevant content, Brian demonstrated the feature to the audience.
“That’s great” said Kim. “I don’t really like the icon you selected for the shopping cart but I know, we’re not going to discuss this now. I’ll take a note for when we discuss the design and graphical interface”.
“Next. As a customer, I want to be able to specify my billing address so I can pay for the product I am purchasing” said Kim.
Once again, Brian clicked on a field, entered some text and press the submit button.
“Exactly” said Kim.
Alessia jumped in. “Why aren’t you also demonstrating the shipping feature at the same time”
“I’m sorry Alessia. Chickens are not allowed to participate in the product demonstration” said Jordan. “I can stay at the end of the meeting to explain our process. Let us continue the demonstration so we can stick to our time box”.
Alessia was shocked. In most meetings, people can interject and ask questions. Remembering how helpful Jordan had been earlier in the week with the coffee machine, she decided not to say anything for the time being but she wondered if being called a chicken was as bad as it sounded.
The product demonstration lasted over 90 minutes with most features being accepted as presented. Despite the fact the her question hadn’t been answered, she felt the presentation had been successful especially based on the constructive team dynamics she had witnessed. Once the meeting was over, she excused herself and walked back to her office.
On her way to her office, Alessia stopped for a coffee. Although it was past 4 pm, she wanted to get a jolt of caffeine to wrap up the week before leaving for the week-end. As she was pouring milk into her cup, Jordan walked in. “I thought I would find you here” he said. “You left before I could answer your question. I hope you didn’t take it the wrong way when I asked for you to wait until the end of the meeting to answer your question”.
“That part, I was fine with. What’s the issue with the chicken” asked Alessia.
“If you have a few minutes I can tell you about that” replied Jordan.
“Sure, come to my office. I have a few questions for you” said Alessia.
As the two walked toward her office, Alessia started “I am impressed with the team dynamic I saw this afternoon. A few months ago, all I heard were complains from marketing and from our team how catastrophic this project was and now, it sounds like everyone is pushing in the same direction. What happened?”
“The short answer is that we started doing things differently. The longer answer a friend of mine had tried a new approach in his organization with a lot of success and after I heard about it I asked Scott to attend a training. I started to apply my learning to this project. It was a disaster anyway when I took it over so I thought I couldn’t make it worst but trying a new approach. So I did.” said Jordan.
“How do you call this approach and most importantly can we send more people to this training?” quickly asked Alessia.
“Yes you can send more people but before you do, would you mind if I asked you a few questions?” asked Jordan.
“Sure” replied Alessia.
“The project management approach I’m using is called SCRUM and it is part of an Agile approach to software development.” said Jordan as they entered Alessia’s office.
“Have a seat” she said.
“Thank you. So what problems are you trying to address?” asked Jordan.
Alessia pointed toward the white board she had used earlier in the week.
“You have a lot of issues” said Jordan “Can I help?”
“Tell me something I don’t already know… Yes, maybe you can help” replied Alessia. “Can we apply your project management approach to our BI project?”
“I don’t know for sure but off the top of my head, I would be tempted to say, yes. Would you like me to look into it?” asked Jordan.
“If you don’t mind, give it some thoughts and maybe we can discuss your ideas next week?” said Alessia.
“Wow, it’s already passed 5 pm! Sure, give me a few days and I’ll schedule a meeting with you next week” said Jordan. “Have a good week-end”.
This blog post presents a preliminary version of the Table of Content of the collaborative Agile BI book. You can access the various sections currently available and you may register to join this collaborative effort to start contributing. I also invite you to comment this content at the bottom of theblog post.
INTRODUCTION
CHAPTER 1 – Dealing with the Problems
CHAPTER 2 – Using SCRUM for the BI project
(References: Agile Project Management with Scrum, Agile Data Warehousing: Delivering World-Class Business Intelligence Systems Using Scrum and XP, Managing Agile Projects)
CHAPTER 3 – Changes and Impacts on the Management Team
CHAPTER 4 – Iterative and Incremental Analysis and Development
(References: Agile and Iterative Development: A Manager’s Guide, Agile Documentation: A Pattern Guide to Producing Lightweight Documents for Software Projects)
CHAPTER 5 – Transforming the Project Reports and Reporting Mechanisms
CHAPTER 6 – User Stories, Estimating and Planning Poker
(References: User Stories Applied: For Agile Software Development, Agile Estimating and Planning)
CHAPTER 7 – Test-Driven Development and Continuous Integration
(References: Test Driven Development: By Example, Continuous Integration: Improving Software Quality and Reducing Risk)
CHAPTER 8 – Database Refactoring
(References: Refactoring Databases: Evolutionary Database Design, Agile Database Techniques: Effective Strategies for the Agile Software Developer)
CHAPTER 9 – Evolving the Corporate Culture – Change Management
CHAPTER 10 – Communicating the Success and the Fundamental Changes
CHAPTER 11 – Project Retrospection and Conclusion
References: (in this section we will maintain a list of books, magazine articles and blog posts used for this book)
This blog post presents a preliminary version of this chapter of the collaborative Agile BI book. You can access the various sections currently available and you may register to join this collaborative effort to start contributing. I also invite you to comment this content at the bottom of the blog post.
The hand on the clock seemed to be moving slower. It was approaching 5:15 pm and Alessia was wondering how much longer this meeting would be. The strategic committee met on a monthly basis and although the board room was situated on the 12th floor overlooking the beautiful downtown area, she knew better than to expect to have dinner with her family tonight.
Dan called up the next management team member to speak. “Vincent, you have 30 minutes to give us an update”.
Vincent was the longest standing member on the management team and although he had the title of Vice-President of Supply Chain Management, very few people would have been able to tell what he actually filled his days with. Vincent always seemed very busy and he had become a master at deflecting situations so they would become someone else’s problem. He was about to use his famous strategy once again.
“Team”, Vincent said. “The economy is hitting our suppliers very hard and without reliable suppliers our entire business is at risk”.
Although everyone knew the coming months would be challenging for the organization, they didn’t need Vincent to point out the situation. Especially not in a condescending fashion.
“I know we all have many projects underway and the quality of the recent hires wasn’t great…” he continued before getting interrupted by Sandra, the V-P of Human Resources.
“Excuse me Vincent, can you please clarify your comment? I believe my department has been offering a great service and this is the first time I hear you complain about our recruiting” said Sandra.
“I’m sorry Sandra, I certainly didn’t mean to imply your team wasn’t doing a good job! Quite the contrary, your team is always available when my people need help. I was referring to IT”.
While Sandra politely smiled, relieved that her department was maintaining a positive track record, Alessia almost spat the sip of water she was drinking.
“What? What do you mean by that Vincent?”, Alessia asked.
“Well, you know. I’ve been waiting for months now to get a report that would show me which suppliers ship on time and which ones don’t. That is critical information for my department and for the entire company but your department doesn’t seem to consider this a priority at all.”, Vincent explained.
Alessia was aware that Vincent had requested the supplier reports months ago but since her headcount had been reduced by 15% in the last 6 months and that Vincent hadn’t followed up after his initial request, she assumed the reports weren’t that important. Come to think of it, Alessia remembered that Vincent had lost a key supplier a few weeks back so she assumed Vincent was trying to deflect the pressure by bringing up an issue with the development team. The V-P of Information Technology didn’t want to leave the meeting with a strike against her for something so trivial, so she decided to take the conversation to a different front.
“Indeed Vincent, those reports are critical to help you properly manage your department which will have an impact on our entire company. We wouldn’t want you to lose another key supplier”, Alessia added to make sure Vincent’s issue was still visible to the whole group before moving on to another issue.
“Guys, we all understand the economy is difficult and we made big sacrifies. My department had to let go 5 people since the beginning of the year so we are short staffed to tackle all the projects that are coming our way…”, Alessia said before getting interrupted by the V-P of Finance who’s attention was taken away from his blackberry for a minute.
“Alessia, we know your department had the biggest cut this year but the average salary of your developers is much higher than those of people in other departments…”, said Mark.
“Mark, let me stop you there. Please let me finish the point I’m trying to make before jumping in but let me assure you that it is not about the decision to reduce my headcount. As difficult as it was, I still support the decision but my resources aren’t being used efficiently.”, Alessia continued.
Mark seemed pleased with his comment and the answer he received and discreetly went back to his blackberry.
Alessia staired in Dan’s direction asking her General Manager for permission to continue. Dan nodded.
“Guys, as I was saying my resources aren’t being used very efficiently. I don’t mind re-assigning resources to the most critical project but we would first need to agree on what it is. As we speak, my team is being pulled in all directions.”, Alessia continued.
“We are trying to move the e-commerce project forward but Marketing keeps changing the requirements….”, said Alessia before getting interrupted by Cynthia.
“We need to respond to our clients’ needs. This is the mission of Marketing. The market is changing so we need to adapt”, added Cynthia.
“Does the market change weekly?”, asked Alessia
Sensing a trap, Cynthia opened up another track to the discussion.
“The last 2 meetings scheduled, your business analyst didn’t even show up. How am I supposed to get my application ready to launch when your team doesn’t even do their part?”, added Cynthia.
“First I hear of this. I’ll look into. It might be because he was working with Greg’s team producing Excel reports”, said Alessia.
“What do you mean?”, asked the General Manager.
“Dan, the Sales’ team has jeopardized my 2 business analysts and 6 developpers to help them recreate their old Excel reports in the data warehouse.”, Alessia answered.
“Greg, can you explain”, asked Dan.
“Of course. As we all know we are working hard to increase the sales but my account managers don’t have the information they need since we moved to the new data cubes”, said Greg.
“Data marts”, replied Alessia. “They are called data marts”.
“Right, data marts”, said Greg. “Since we moved to the data mart, my team is missing the information they need to do their job so I asked them to work with IT to get their old Excel created in the data mart”, added Greg.
Dan looked at his watch and said “We still have 6 items to cover on the agenda and it’s getting close to 6 pm. I’m asking Alessia to sort out this mess and report back to the team at the next management committee meeting next month”.
Dan looked at the agenda and said, “Next item is for you Richard. How are we doing with the upcoming golf tournament?”.
Alessia phased out for a few minutes. She was getting really tired of how things were going but she couldn’t think of a solution at this point in time. She decided to pay attention to the remainder of the meeting for now and would try to resolve the issue tomorrow.
This blog post presents a preliminary version of this chapter of the collaborative Agile BI book. You can access the various sections currently available and you may register to join this collaborative effort to start contributing. I also invite you to comment this content at the bottom of the blog post.
Alessia had woken up early that morning, her head still ringing from the meeting she had the previous day. She decided to work out before going to the office allowing herself some time to put together a plan of action before getting to the office.
The preliminary version of Chapter 1 is now available. It is incomplete but I am sharing it with everyone so you can get an idea of the tone and direction I would like the book to take.
Let me know your thoughts…
Sign up to join our collaborative book writing project.
The preliminary Table of Content of our Collaborative Agile Business Intelligence Book has been made available.
You can read about our Collaborative Agile Business Intelligence Book initiative and sign up to join our team.
My apologies to everyone who tried to access the registration form to sign-up for the collaborative book. The issue has finally been resolved and collaboration can now begin!
Recent Comments