Analytical-Mind

In their recently released study “Agile Development: Mainstream Adoption Has Changed Agility“, Forrester reports that “35% of respondents stated that Agile most closely reflects their development process”. The report is based on Forrester’s/Dr. Dobbs Global Developer Technographics Survey, Q3, 2009, which surveyed 1298 application development professionals.

In the past few years, Agile processes have not only gained increasing adoption levels; they have also rapidly joined the mainstream of development approaches. And while more organizations are adapting to Agile conventions, Agile is also adapting to the workplace. Perhaps the clearest sign of the mainstreaming of Agile is the abandonment of orthodoxy: Teams are puzzling out the mix of methodologies and combining them to fit within their organizational realities, blending Agile and non-Agile techniques and practices to create a hybrid methodology that fits larger organizations. Other changes, such as new team dynamics and the redefinition of roles such as the business analyst, show the genuine force behind Agile adoption. It’s time for software development professionals to stop sitting on the fence where Agile is concerned. According to those who have successfully adopted Agile, the benefits are well worth the effort, and with the recent dramatic increase in Agile adoption, the probability of working in or with an Agile team has increased for everyone.

The report looks at Agile Adoption from the following perspectives:

• Agile Adoption Goes Mainstream
• Teams Are Changing To Support Agility
• What it means
• Agile Becomes Mainstream — But Not Without Some Changes

And then offers a recommendation to get the most out of the transition: “App Dev Professionals Should Blend Agile Methods To Meet Their Own Needs“.

Agile methods encourage more-collaborative development than do traditional approaches, and many developers who have shied away from formal development methods in the past — believing them to be the province of “management” — have embraced Agile as a “formal” development process.

The report shows that of the various methodology, Scrum leads as the most adopted Agile methodology with 11% of the organizations reported using this approchach.  Other Agile approaches used are: Agile Modeling (6%), Feature-driven development – FDD (3.8%), Test-driven development – TDD (3.4%), eXtreme Programming – XP (2.9%), Lean development (2.1%), Microsoft Solutions Framework – MSF for Agile (1.8%), Agile Data Method (1.6%), Adaptive Software Development – ASD (1.3%), Six Sigma (0.9%), Crystal (0.3%), Behavior-driven development – BDD (0.2%) and Dynamic Systems Development Method – DSDM (0.2%).

When it comes to selecting an Agile methodology, Scrum is the overwhelming favorite – claiming it is simple, practical, and popular.

The report shows that when organizations select a development approach, they do so in the context of their organizational priorities and characteristics. Each organization carefully selects the approach that will best address their weaknesses rather than implementing Agile methodology for its own sake. In addition, participants considered the engineering processes to be critical to the success of the transition.

Teams do not usually implement all of these techniques simultaneously; even in the most mature Agile adoptions, teams pick the techniques that work best for them (…) This variation in the adoption of Agile components indicates that teams are more concerned with making sure they are working well together and producing high-quality software than with changing their software engineering process.

Perhaps the most important aspect of Agile’s entrance into the mainstream is the way that teams pragmatically mix methodologies. Instead of sticking strictly to a particular Agile orthodoxy, teams cherry-pick Agile methods, often including non-Agile techniques in the mix as well

Overall, the report clearly shows that organizations who have transitioned to an Agile approach use a pragmatic strategy and adapt the methodology that are best suited for their environment and context. While only 27% stick to a particular methodology, 63% mix different methodologies or mix Agile with non-Agile methodologies.

In conclusion

Agile adoption is a reality. Organizations across all industries are increasingly adopting Agile principles, and software engineers and other project team members are picking up Agile techniques. While historically, management has owned “process,” the adoption of Agile methods has pushed ownership into the hands of team members — many of whom have traditionally been skeptical of process and methodology. Broad Agile adoption requires careful consideration; a strong Agile adoption strategy should include:

• A support plan. Adopting Agile practices is not a trivial change; it requires support and time to become effective. The use of external coaches, training materials, and internal support groups can greatly increase the speed and success of adoption.
• Flexible adoption models. One size does not fit all.
• A focus on team empowerment. It sounds really easy, but this is about more than just saying that teams are empowered.
• A tool strategy. A single team in one location working alongside a customer may be able to work without any electronic tools, but as organizations scale and teams become more distributed and part of much larger releases, Agile methods benefit greatly from tools.

Related Research Documents

• “From Agile Development To Agile Engagement” - May 6, 2009
• “Ensure Success For Agile Using Four Simple Steps” - March 24, 2009
• “Make Agile Lean To Boost Business Impact” - December 17, 2008

Like most modern Homo sapiens, when you hear Agile Leadership or Agile Management, you think of:

• [if you are outside the business world] A business-person who can use a combination of balance, coordination, speed, reflexes, strength, endurance and stamina to achieve his objectives;
• [if you are inside the business world but outside the information technology field] A person who has the capability of rapidly and cost efficiently adapting to changes in an attempt to deliver on his objectives;
• [if you are inside the information technology field] A person who manages a software development team who uses methodologies based on iterative development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams with the objective of delivering value.

I apologize if you are outside the business world because this is not the perspective I wish to cover. For people in the other two categories, you are partially right.

I attempted to define Agile management and see how I could apply Agile principles to management a while ago and since then, I have been able to piece the puzzle together. Agile Leadership requires less technical knowledge than its cousin but it heavily relies on the same principles.

A high level view of the model

If you have been reading my blog for a while (thank you!) and even if you haven’t, you will realize that I have been covering various parts of this model already:

People: The people dimension covers all aspects of competencies, motivation, culture, collaboration and communications that enable the organization to achieve its business objectives. While every effort is directly or indirectly related to people, this perspective focuses primarily on the ability of individuals to contribute to the achievement of objectives.
[related tags: 360-degree feedback, coaching, collaboration, community, decision making, feedback, leadership, management, organizational structure, people management,servant leadership]

Processes: The process dimension aims to define the working methods and approaches to be followed in carrying out tasks in line with the overall objective of delivering business value.
[related tags: agile, agile management, scrum]

Tools: The technology dimension covers the various tools and technologies that support the organization in achieving its business objectives.
[related tags: none, I haven't covered this dimension]

Value: The value dimension covers the business capacity to effectively deliver value within the appropriate time. The delivery of value is the fundamental purpose of the organization.
[related tags: ROI]

As you can see, I have mostly covered the People dimension of the model while I have purposely left the Tools section un-covered. The reason for this is that there are already thousands of web sites on the topic of Agile and technology.

In an upcoming series of blog posts, I will present a more detailed perspective of what Agile Leadership truly means based on our experience. Stay tuned…

On the importance of creating the right organizational culture (Thanks to Andrew)

By the time we got to 100 people, even though we hired people with the right skill sets and experiences, I just dreaded getting out of bed in the morning and was hitting that snooze button over and over again Corner Office – Tony Hsieh of Zappos – Celebrate Individuality – Question – NYTimes.com.

On why an Agile approach is better suited to deliver value (Thanks to Alfonso)

Most organizations that depend on software are struggling to transform their lifecycle model from a “development” focus to a “delivery” focus. This subtle distinction in wording represents a dramatic change in the principles that are driving the management philosophy and the governance models – Improving Software Economics

On the meaning of Agile transformation for managers

What many people mistakenly do is equate agile project management with doing more work, with less documentation and fewer people. Although the premise is to get more done in a more favorable way, I have never met a team that could successfully implement agile principles without having to slow down first - VersionOne – Agile Adoption For Managers.

On the fact that the true value of an organization is not mapped via its organizational chart

But it’s not the fact that you have many more boxes and lines that I’m most envious of.  It’s your “white space” I want – Oh, Yeah? Well, My Org Chart is Bigger and More Beautiful Than Yours!

On the need to manage self-organized teams when required

The interesting thing is, the further we go into agile management territory the less typical the managerial job we expect. Teams are self-organizing and cross-functional, and sometimes we think a manager should just get out of the way. By the way, surprisingly often this is exactly the best choice. But whenever one of the asshole-moments is needed, it is time to show up and do what has to be done. Otherwise the atmosphere starts rotting as people wait for someone who will fix things. Someone who will do something about this guy adding a new technology every time he reads some nice article. Someone who will deal with that lass taking a few days off because she doesn’t really care about the project being late and the team working their butts off to get back on the right track. That’s always a job for a manager, and a harsh one, no matter how self-organized the team is - Good Managers Sometimes Have to Play Assholes – NOOP.NL.

“Americans hate their jobs more than ever” … ”Majority of Americans dislike their jobs” …

These are only 2 examples of a quick google search that returned over 44.1 million pages. Try “I hate my job” and the content of the pages returned is also very sad.

I don’t intend to go into socio-psychological analysis in this post but I wonder if something as simple as trusting your employees to do their job properly would actually increase job satisfaction?

For most people, enjoying their job would simply mean doing the same type of work but in a different work setting. Many people have spent years studying to develop their expertise in a specific field that they love. Then, one day, they start working and life becomes miserable – not because they hate what they are doing but because of the way the are treated at work. Once again, I don’t want to go into harassment or this type of treatment. The only point I’m raising is that letting professionals determine the best way for them to complete their work would is such a simple of increasing job satisfaction.

“Yes, but I’m the boss” – you reply.

So what? The fact that you were hired to lead or manage people in achieving a team or departmental objective doesn’t make you the most qualified individual to resolve day-to-day issues.

“Yes, but I’ve done this job before” – you insist.

Once again, so what? The individuals performing the job now bring different skill sets and expertise to the equation and as such are qualified to address their work as they see fit. You may provide guidance or answer your employee’s questions when they come ask for help but not tell them how to do their job.

Put together a community of expertise so people doing similar work can support each other. Provide tools if they need, support your employees in finding the right answers to their problems but don’t tell them how to do it.

There is a small Japanese car-manufacturer that understood that concept a while ago. They are now the largest car manufacturer in the world. Don’t you wonder how they achieved their success?

The problem with slow feedback loop

I recently re-read The Fifth Discipline: The Art & Practice of The Learning Organization. Peter Senge talks about the impact of feedback loops in individuals and organizations’ learning process. Feedback loops (aka retrospection) are also critical in Scrum and the Agile approach.

What does this have to do with my plant you ask? Simple. I was the victim of a slow feedback loop myself and it almost killed my wife’s beautiful Azalea.

A few weeks ago, I bought my wife a beautiful Azalea for our anniversary. It was in full bloom. It was simply beautiful!

In addition to looking very nice, the salesperson at the flower shop told me it was low maintenance. This is a key feature for us since we usually don’t do very well with house plants. I bought the plant and took it home.

Following the instructions, we would add water every few days and made sure the plant didn’t have too much direct sun light. One morning, we noticed some of the beautiful flowers were starting to dry out. Experience tells us that when something is dry, you add water – so we did. We increased the frequency of the watering ritual from once every 4-5 days to once every 2-3 days.

Much to our surprise, the situation didn’t improve. Actually, it was even worst, more flowers were drying out – so we thought, let’s add more water. We moved the plant next to the kitchen sink so we would remember to add water every morning when preparing coffee. A week passed by and the results got worst. In addition to dry flowers, the plant was loosing all its leaves. It started to look pretty bad.

While I was preparing coffee this morning, I went to add water to the plant only to discover that the plant was immersed in water. There was so much water that it covered the earth in the pot! S*%t! We are drowning the plant!!

Then it hit me. Because we weren’t getting any obvious feedback, we assumed what we were doing was good and we continued until it was almost too late. Had we had indications sooner that our actions weren’t the right ones, we could have changed them and possibly address the right issue.

This is exactly what we teach our customers and what Peter Senge was explaining. Without rapid feedback loop, it may take a while for people to realize they have been doing something wrong.

Agile Leadership Lessons *

I came across an article that was published 6 months ago in CIO magazine. In the “7 Agile Leadership Lessons for the Suits”, the author makes interesting parallels known concepts and Agile leadership. Below are the 7 lessons and a brief summary of what each of them entails.

1. Trust the Wisdom of Teams: Software is a result of a team effort, so a manager’s main concern should be nurturing the team.
2. Even Self-Organized Teams Require Coaching: it would be naive and irresponsible to imagine that turning teams into self-organized machines means abdication from managerial duties. 
3. Adapt or Get Out of the Way! The New Manager’s Role: collaborative environments, trust, transparency of information, and building productive and sustainable teams.
4. Motivate, Don’t De-Motivate: Appraisals, Bonuses and Compensation: everything you are doing in your HR functions is wrong, because it is supported only by our illusions, not by facts.
5. Write Value-Based Vendor Contracts: that clients are tired of vendors who promise low-ball prices and then make their money on change requests.
6. Avoid Building Plank Roads: Waterfall was a plank road, said Poppendieck, aimed at solving the process problem by applying project management methods already used in other industries. But its applicability to software development was dramatically overestimated, and the ramifications of this strategic mistake haunt us to this day. 
7. Who Do You Trust?: We are all guilty of stereotyping when we interact with others. We subconsciously label people.

If anybody gave you the impression that Agile was easy, think again. You may want to do some research and determine if the benefits will exceed the pains for your organization before jumping in.

I'm facing a big problem. Can you help me? *

There are questions and challenges that keep coming up. No matter if you work on the business side or the technical side of your organization, you have certainly faced or might be dealing with some of these challenges and you aren’t quite sure what to do. If the statements below sound familiar to you, you may want to read on.

• My project team constantly misses deadlines.
• The project team keeps exceeding its budget.
• My project team doesn’t deliver on the requirements.
• The end users don’t know what they want.
• The requirements keep changing and that constantly impact our project plan.
• The project team develops software components that don’t seem to have any business value and they seem to produce more paper tha software.   
• My team develops software that doesn’t satisfy my users.
• The project team usually finds problems very late in the development process.
• The project team does not have the right skills.
• The project team is tired, nobody is having fun and we are losing good people.
• I need to wait a long time before the project team gets me the information I need.
• We know we have issues but we don’t know where to start.
• We need to outsource our software development activities in order to cut costs.
• The project team delivers poor quality software.
• We have started using Agile for a small project and our management team wants us to scale it to the rest of the organization.

Agile principles can work for your Data Warehouse / Business Intelligence project but it is critical to determine which business problem you are trying to solve. Below, we are presenting a list of issues frequently encountered and we offer potential solutions to each of them.

What is your challenge?

My project team constantly misses deadlines.

How Agile can help

Implementing Scrum as your project management approach and the proper reporting tools (such as burn down charts) would help you anticipate potential delays and address your delays in a timely manner.

In addition, using frequent face-to-face communication instead of communicating through a project plan will increase your team’s productivity, performance and compliance to defined time lines.

What is your challenge?

The project team keeps exceeding its budget.

How Agile can help

The situation might be worse than you think because you may also be delivering the wrong functionality to your users.

Using long range planning often results in high variance in the actual budget being spent whereas delivering software in short iterations (up to 4 weeks) will allow you to better monitor your costs while ensuring you deliver the expected results.

What is your challenge?

My project team doesn’t deliver on the requirements.

How Agile can help

This could be two different issues – the requirements aren’t well defined by the end users or the requirements aren’t clearly understood by the development team.

In either case, enforcing close collaboration between the end users and the development team by co-locating the team and enforcing face-to-face communication will greatly increase the chances of delivering the right software.

You could turn the situation around by implementing the right processes which would allow you to welcome changes during the development phase and adapt to your changing business challenges.

What is your challenge?

The end users don’t know what they want.

How Agile can help

This is not unusual but it is not necessarily an issue with your end users. With most projects using a traditional software development methodology, end users are asked at the beginning of the project for their requirements and will eventually see the results months or years later once the software has been delivered.

With an Agile approach, end users are asked to define their short term requirements (for the next 4 weeks) instead of defining the entire scope of the project which will help the project team to deliver on the requirements.

What is your challenge?

The requirements keep changing and that constantly impact our project plan.

How Agile can help

Indeed, if you are using a traditional software development methodology you are likely to run into changes as the market dynamics evolve frequently. With an Agile approach, your team will not only learn to deal with changes in requirements but will even learn to embrace those changes and adapt accordingly.

What is your challenge?

The project team develops software components that don’t seem to have any business value and they seem to produce more paper than software.   

How Agile can help

Why are you? The focus of your development team should be to deliver value – working software. If for some reasons, a lot of your development team’s time, efforts and energy are spent working on documents (project plans, requirements, architecture, models, etc.) instead of working on software you probably need to reconsider the software development approach you are using.

Using a pragmatic and realistic approach like agile for your software development process will address the most critical problems typically encountered by a software development team and will greatly increase your return on investment.

What is your challenge?

My team develops software that doesn’t satisfy my users. Satisfying the requirements of your users is critical for a software development team and applying agile practices will greatly help.

How Agile can help

The integration of the business users as part of the development team is a great way to user their requirements are properly addressed. In addition, using techniques such as user stories that describe the features from an end-user perspective makes it easier for the development team to meet the requirements once it is clearly documented.

What is your challenge?

The project team usually finds problems very late in the development process.

How Agile can help

It is frequently recognized that the later you find problems in the process, the more expensive it becomes to fix them. Based on that conclusion, the implementation of an Agile software development approach forces the team to present the outcome of their labor to their users in order to demonstrate and test the software frequently which bring to the surface the issue.

What is your challenge?

The project team does not have the right skills.

How Agile can help

Although it may be possible that your team lacks some skills, it is unlikely that your people aren’t qualified for your project.

Your team might need some help and coaching around some specific software engineering practices and with the right coaching, they can improve their development skills.

What is your challenge?

The project team is tired, nobody is having fun and we are losing good people.

How Agile can help

It has frequently been proven that motivated employees deliver better results so if you want your project and your team to succeed,creating an environment where people can learn, work on challenging project and feel they are contributing to the success of the organisation will greatly help.

What is your challenge?

I need to wait a long time before the project team gets me the information I need.

How Agile can help

The business user and the development team may not be working closely together and you are probably using a traditional waterfall approach where the development team asks for requirements, goes off to develop and test the queries and reports and once they feel comfortable with the results will present them to the business users.

By including the business users within the project team and using an incremental development cycle, you would see the progress being made on your original request and could influence the amount of details required. Keeping small iterations and short feedback loops would allow you to quickly see how the development team is doing and quickly get access to the information you need.

What is your challenge?

We know we have issues but we don’t know where to start.

How Agile can help

An agile approach helps improve communications within the development team and with the business users. Having frequent demonstration of the system’s capabilities will quickly bring to the surface the issues and will allow the team to address them.

What is your challenge?

We need to outsource our software development activities in order to cut costs.

How Agile can help

Not necessarily, a better approach might help you reduce your costs. The Standish Group Study (Reported at XP2002 by Jim Johnson) showed that as much as 64% of system functionality was never used. Using an Agile approach to build the right software will immediately reduce the overall cost of developing software within your organization.

What is your challenge?

The project team delivers poor quality software.

How Agile can help

You may want to consider implementing test-driven development (TDD) where developers need to start by writing their test before developing new code or you may want to leverage pair programming.

In addition to changing the way you develop software, it would be beneficial to use short development cycle and demo the resulting products frequently in order to gain visibility to the software and address the issues in a timely fashion.

What is your challenge?

We have started using Agile for a small project and our management team wants us to scale it to the rest of the organization.

How Agile can help

Absolutely. Now that you have seen first hand the tremendous benefits of using Agile practices, you may want to coaching and guidance to help you roll out the new approach to the rest of your development organization and take full benefits at a larger scale.

* Picture by Mikey G Ottawa used under the Creative Commons (CC) agreement. The view expressed in the blog post is the one of the author. The photographer does not endorse in any way the content of this blog post or the work of the author.

I wanted to summarize Tuesday morning’s session at the TDWI BI Executive Summit.

Larissa Moss (President of Method Focus) presented “Agile Development: What is it and will it work for BI?” where she explained that after failing at a DW project she was working on using the waterfall approach, she stumbled into Agile in the mid ’90s. After explaining what Agile actually is and how it differs from a traditional waterfall approach, she asked the “$64,000 question“, “Can Agile work for BI?” to which she answered “It depends what you call BI“.

Larissa argues that Agile BI works well for companies that don’t have to deal with the complexity of the data (i.e. don’t deal with building an underlying data warehouse). As such, she explained that using an Agile BI approach to do a data warehouse project where the development team has to deal with end-to-end solution including data standardization, data modeling and ETL cannot work due to the complexity and the un-known associated with the underlying data.

Larissa finished her presentation by proposing a solution to address this situation – Extreme Programming^TM which is a 16 steps Agile methodology she developed to address the pitfalls of the traditional waterfall approach. Her book called Extreme Programming^TM is scheduled to be released shortly.

Jim Hill (Director of Data Management at 1-800-contacts) then presented “Applying Agile Techniques to BI” and how his organization has been successful for the past 4 years in using Agile Development “for all software development projects and BI activities“.

Jim explained the process used by his business users to log their requests (user stories) which are then prioritized weekly and included in weekly iterations. He gave an example of 1-800-contacts’ Call Center BI which is deployed to over 400 agents’ desktops and for which the data is refreshed every 15 minutes.

Following Jim was Wyatt Weeks (Group Manager Business Intelligence at Sport Authority) who presented “Agile Data Warehousing at Sports Authority”. Wyatt told the audience that his team had chosen to use Agile Development for the first time on a “Large, high risk project with short time lines” and following their success, the team has continued to use Agile to deal with changing priorities.

Wyatt’s Scrum team consists of 7 to 9 members and all phases of their BI projects (architecturee, data modeling, ETL, and report development) are delivered through monthly sprints. He highlighted the benefits of Scrum: team collaboration, enforces communication and visibility, and clear definition of priorities. 

The 3 speakers gave good presentations and agreed that indeed Agile Development can be used for BI projects as long as it is adapted to the reality and constraints of the organization.