Gartner studied the market and attempt to predict trends in their latest report: Predicts 2010: Agile and Cloud Impact Application Development Directions.

As organizations seek to improve productivity and reduce application operating and maintenance costs, we will continue to see an evolution of software development tools, platforms and practices. To take advantage of this, organizations must shift structures and practices while embracing new technologies — a challenging proposition.

Gartner’s analysts (Thomas Murphy and David Norton) predict that by 2012 “agile development methods will be utilized in 80% of all software development projects”. The authors explain that although Scrum will continue gaining in popularity over the coming years, organizations will not be successful in their transition unless they move toward a team-focused culture. As was mentioned in their previous report, very few organizations use a pure-Scrum approach and most rely on an hybrid approach (waterfall and Agile).

The report highlights that organizations struggle to implement true collaboration in the context of globally distributed teams. A situation that has amplified in recent years with outsourcing and off-shoring of software development projects.

In the other hand, the report confirms that teams who have successfully moved to Agile do see productivity improvements especially in “the flexibility of the development team to respond to shifting requirements”. This is especially true for web-based application developments where rapid responses to a changing environment is critical.

The authors point out that organizations need to properly invest in such a transition in order to achieve success.

Organizations that do not make use of key agile practices and do not invest in training and supportive tools’ infrastructure will find that a shift to pseudoagile, while potentially delivering a short-term productivity bump, will end in long-term declines in quality and productivity (…) the promise of four times the improvements in overall productivity has been and will be achieved by select organizations.

Gartner’s report highlights that “development organizations have been making a shift toward agile methods, but this is still slow to move beyond development, and often is a mixture of waterfall practices utilizing an agile or iterative project cycle”. The authors also recommend to “look for opportunities to utilize agile development practices, but recognize that it requires changes and commitment on the part of business and IT”.

Gartner concludes with a few recommendations to help organizations maximize their return from an Agile transition.

• Recognize the cultural changes that are at the heart of agile.
• Don’t allow agile excitement to drive cowboy-coding practices.
• Agile requires discipline.
• Recognize that scrum is only a partial solution, and focus on a collection of practices.
• Find tools that enable collaboration and help automate repeatable, consistent practices.

Related documents:

• Don’t Let Short-Term Agile Create Long-Term Pain
• The Current State of Agile Method Adoption

As a follow up to my previous post, this second post in a series of 4 short articles written in collaboration with my colleagues Stéphane Lécuyer, Jean-René Rousseau, Sylvie Trudel, Joël Grenon, and Eric Laramée, addresses the impact an Agile transition typically has on some of the traditional software development roles: the project manager, the architect, the business analyst, and the QA specialist.

Based on our experience, here are possible strategies to properly map the traditional roles to the three roles defined by Scrum.

The Project Manager

Traditionally, the project manager is responsible for determining who, what, and when activities need to be performed and then to ensure the team complies with the plan that was prepared to meet the budget, time and scope constraints.

With the traditional approach, project management is based on compliance with the plan while Agile and Scrum propose a different approach where maximizing the business value is the main vector of project management. Under this new approach, the product manager needs to collaborate with the team members and delegate to them some of his traditional responsibilities since they will determine who does what, and when within the constraints of the project.

In this context, the role of the Scrum Master is to enforce the process and seeks to build an efficient self-organized team. To the question “do we still need a project manager in Agile?”, experience shows us that in most organizations, the answer is yes.

The need for accountability, regulatory compliance and alignment with the framework and IT governance are not covered by the role of the Scrum Master and as such remain the responsibility of the project manager.

However, the project manager needs to adapt its management style and use leadership rather than authority with the team to get things done. In the context of a multi-team organizational structure, the presence of a project manager is also valuable, where he is coordinating the teams and the synchrony between them and between entities external to the project teams.

From our experience, some project managers are more willing to become product owners while others will feel challenged by the role of Scrum Master. In the end, it will be the responsibility of the organization to determine how to redefine the roles and their associated responsibilities.

The Architect

Similar to the project manager, the architect is known to play a different role post-transition compared to that required in traditional development teams. He must act as a consultant to the teams and provide them with the necessary support instead of dictating the rules and guidelines to be followed. The architect should also be familiar with the concepts of emerging architecture, where just enough architecture is planned to allow the team to innovate and find the optimal solutions.

The architect then acts as a catalyst for sharing best practices within the organization. Here is a list of practices summarizing the changes of behavior for the architect:

• Is an active member of the development teas, helping to build the right software and acting as consultant;
• Does not attempt to predict the future, he provides a coherent vision but knows that tomorrow’s problems will be easier to solve tomorrow;
• Is changing its architecture in an incremental way, leaving room for emergence;
• Does not seek to document everything to perfection, he focuses on a few relevant diagrams and documents the best practices based on his audience;
• Seeks to validate its concepts through concrete experiences.

Once again, although the role of the architect does change after an agile transition, it remains an important role to be filled.

The Business Analyst

The business analyst is another role that seems neglected by Scrum. To ensure close collaboration between the team and the Product Owner, Scrum ensures that the necessary elements are effectively communicated directly to the team without a formal and complex documentation. However, to ensure continuity of information, we know that functional documentation that is adequate and representative of the software to be developed is essential.

The business analyst becomes a valuable contributor to the Product Owner. The responsibilities of the business analyst are as follows:

• Supports the Product Owner in gathering and writing the required stories;
• Does just enough analysis for the functionality to be carried out during the next iteration;
• Prepares and updates documentation used at the end of each iteration;
• In collaboration with the QA Specialist, helps determine the quality assurance strategy.

In a multi-team context, the business analyst may be called upon to play the role of Product Owner. He then becomes responsible for core components of the product within the various sub-teams.

The Quality Assurance Specialist

Quality is a fundamental concern in Agile project management and each iteration should produce an increment of quality software. To do this, we recommend incorporating a quality assurance specialist within the Scrum teams, and right from the start of the project. A QA specialist assigned to a Scrum team has the following responsibilities:

• Participates in planning sessions to raise issues relating to quality;
• Helps clarify the definition of “Done”‘;
• Prepares plans for acceptance testing;
• Validates the increments of product delivered.

Other Roles

As will be presented next week in “Part 3: Impact on the functional and people managers”, managers also get impacted by an Agile transition.

If you have been reading about Agile for a while and are interested in a transition or if you have already initiated a transformation, you have previously heard all the benefits that Agile can bring to your organization but …

Are you aware of the impacts such a transition will have on your organization? On your team? And on yourself? Would you know how to deal with these impacts?

If you believe that implementing Agile within a company simply means reducing documentation, standing up during daily meetings, using whiteboards and post-it notes, and getting rid of the project manager, you will certainly be shocked to see how profound the changes can be.

In a series of 4 short articles written in collaboration with my colleagues Stéphane Lécuyer, Jean-René Rousseau, Sylvie Trudel, Joël Grenon, and Eric Laramée, we aim to highlight some of the most common (and rarely described) impacts an Agile transition can have on an organization. The articles will be published weekly and will cover the following 4 impacts.

• Part 1: Impact on the organization
• Part 2: Impact on some of the traditional roles
• Part 3: Impact on the functional and people managers
• Part 4: Why a coach is useful

Adopting Agile practices is not a trivial change; it requires support and time to become effective. The use of external coaches, training materials, and internal support groups can greatly increase the speed and success of adoption. - Forrester Reports “Agile Development: Mainstream Adoption Has Changed Agility”.

Many organizations rely on external consultants to help them successfully transition to Agile. Others initiate a small transition after having researched the best practices. Having gained experience from the implementation of Agile within organizations over the last 8 years, we can attest that the impacts related to the establishment of an Agile development approach affect many areas in the organization. Through our experience, we have prepared a high level description of potential impacts you may want to anticipate before getting deep into your transition.

Next week’s post will address more specifically to impact on the role of the project manager, the architect, the business analyst, and the QA analyst.

As part of a market research for one of our customer, I came across this report published by Gartner in December 2008.

As the pace of agile adoption increases, development organizations must understand the different levels of agile maturity. CIOs and product and development managers need to assess where they fit on the maturity scale, and which level offers the biggest return in their organizations.

The report presents the 6 levels (from 0 to 5) of Gartner’s Agile Maturity Model and corrects a few myths.

Agile adoption and penetration rates are being overestimated. Although the number of companies that are adopting agile practices is, indeed, reasonably high, most organizations use agile in a very small percentage of their overall work.

An agile maturity framework is necessary to help make the case for adoption, process improvement and benchmarking.

Current adoption rates for agile and iterative methods are between 15% and 25%, when taking into account penetration and maturity, with waterfall still the dominant approach. The pace of agile adoption is increasing.

The report concludes that :

As part of an agile readiness assessment, IT development organizations should access their current agile practice maturity at technical, project management and organizational levels. Practices should be assessed for, among other things, their effectiveness and adoption levels in the organization. Adoption should follow initial pilots, and should normally be Level 2, with the aim of developing a consistent set of agile practices at Level 3.

RECOMMENDED READING

• Maturity Model Overview: Application Organizations

Another monthly update on the books I read during the past month. For a complete a list, you can visit my virtual bookshelf.

Coaching

As part of my coaching training, I have purchased this recommended book. The book is a great introduction to what it means to be a coach. It explains how to be an effective coach and provides an approach that can be used for various types of coaching.

Coaching for Performance

My Rating: 

A few words on the book: This is an introductory book to coaching. It provides enough material for people managers who wish to improve their management style by using a coaching approach without getting into too much details. Although the author frequently refers to sport, many of the examples provided and suggested approach do apply to a business context. The book is easy to read and pragmatic but it isn’t enough to completely change one’s management style.

Leadership

Pleased with Collins’ previous books (Built to Last: Successful Habits of Visionary Companies and Good to Great: Why Some Companies Make the Leap… and Others Don’t), I decided to read the third book of the trilogy.

How The Mighty Fall: And Why Some Companies Never Give In

My Rating: 

A few words on the book: Contrary to the previous two books, “How the mighty fall” comes across as an hudge-pudge of simplistic conclusions derived out of the same series of data used for the previous books. Similar to his previous books, Collins introduces a 5 stages model to explain the failures. The problem I had with the book is that the conclusion do not seem to be based on lengthy analysis but on quick conclusions that are company specific – i.e. if a similar company applied the same actions, would they actually get to the same results? It seemed to me that the failures were related to many circumstances that were specific to the organizations – not at a macro level but within the organizations. In addition, the author put too much emphasis on the leader of the organizations and very little on the inner workings.

Meditation

A friend of mine had told me about Deepak Chopra a few years ago and one day I stumbled upon this book. It seemed interesting …

The Spontaneous Fulfillment of Desire: Harnessing the Infinite Power of Coincidence

My Rating: 

A few words on the book: I must admit, I didn’t finish this book. The final chapters are exercises to help improve the SynchroDestiny and that was not what I was looking for. On the other hand, I enjoyed the “scientific” explanation behind SynchroDestiny. I realize that if you are – like many – a fan of Chopra, you will probably enjoy the book. For my part, I am still trying to figure out what was the “destiny” I was supposed to achieve by purchasing this book…

A few days ago, we invited representatives from a potential customer over to visit our office. They are seriously considering a transition to Agile but some of the managers had questions with regards to what an Agile work space could look like. The potential customer is a large insurance company and like most insurance companies, people working there are used to a traditional (very traditional) work space. We could see they had some reservations about the open-concept before coming for a visit.

Their visit lead me to wonder what other Agile work spaces could / should look like, so I came up with the idea of launching a friendly contest…

The “Best Agile Work Space” Contest

I invite you to email me a picture of your Agile work space (martin [at] analytical-mind.com). In the spirit of sharing best practices and getting ideas from each other, I will post your pictures and your company’s name for people to get inspired. You can also share with everyone what makes your work place the Best Agile Work Space. We’ll even ask people to vote!

Let the contest begin to determine the “Best Agile Work Space“. Tell your friends to email their pictures.

Examples of other Agile Work Spaces found on the web

Windows are often a scarce commodity and are doled out to an organization’s favored employees. One of the nice things about an open workspace is that windows are shared. Even if the view is only of our parking lot and can only be seen across three messy desks, at least I can see the window and some natural light - The Ideal Agile Workspace | Mike Cohn’s Blog – Succeeding With Agile®.

Our New Agile Workspace - Our New Agile Workspace on Flickr – Photo Sharing!.

I started to respond in his comments and then remembered that it would be better to capture our workspace on video to share with others.  I am hoping other agile shops will do the same.  We are always eager to see how others are doing things so we can continue to improve - Attempting to Achieve the Ideal Agile Workspace | Derek Neighbors.

Ward Cunningham among others was a big influence early on in making it happen.  The patterns & practices team workspace is optimized for agile development practices.  The workspace features writeable walls, configurable workspace, speaker phones, projectors, focus rooms, and a customer room - Shaping Software » Blog Archive » Microsoft patterns & practices Agile Workspace Tour.

If at any point while you read this post, you disagree with any of my statements, go ahead and click the “Leave a Comment” link. Express yourself!

While the original title of my post was “Why most software development managers need a leadership coach”, I changed it to “Why most managers need a leadership coach” because the situation I have witnessed in the software development industry is also present in many others specialized fields of expertise – at least that’s what many of the people I speak with confirm. Nonetheless, in order not to generalize my assumptions (yet!), I will share my assessment of the people management and leadership capabilities within the software development industry. Let’s begin…

Are you familiar with such problems?

• Unhappy employees leads to high turnover
• Communication challenges
• Employees fail to deliver on their commitment
• People who fail to deliver what they promise
• How To Deal With Problem Staff And Poor Performance

These are only a handful of typical problems encountered by a manager and for most experienced managers, they may sound trivial. Considering that new leaders are not born with management abilities, how can we expect them to be successful in their role?

People managers lack the basic skills

Here’s why I believe most software development managers (and many others) need coaching to become successful in their role (and apparently, I am not the only one who believes this is a valid suggestion). My logic goes as follows:

• Managers – including software development managers – are people;
• There are 2 ways to become successful at something. Either you learn through education or you possess above average intuition and intelligence and can figure out how things need to be done;
• Most software development managers have a technical training /education (examples can be seen here, here, here, and here);
• In addition to their education background, most software development managers mostly played technical roles (software developers, business analysts, application architect, etc.) in their career prior to getting promoted to a management position;
• Most people management positions are complex and require knowledge and experience outside of technology such as Business, Leadership, People Management, Organizational Development, or Psychology;
• Very few people in people management positions have all the requirements (see previous bullet);
• Without prior education and experience outside the software development sector, most managers are ill-equipped to successfully perform in their role.

Coaching is a solution

With an average salary¹ of $85,000 to $125,000 depending on the number of years of experience and location, why wouldn’t an organization invest a few thousands of dollars to hire a coach in order to help develop the people management and leadership abilities? Despite the economic downturn, I still see organizations spend thousands of dollars on training or conferences. Although I don’t argue the value of such events, I doubt they support the development of people management and leadership abilities.

It seems to me that we need to help those in management position succeed. Otherwise, the performance of the entire team will suffer.

Not convinced?

Others seem to agree with this new trend…

1.- Sources:

• The 2010 IT Salary Guide
• Monster.ca Salary Wizard
• Payscale Salary Survey

Agile consultant

You can find more information on our web site.

“Pyxis helps software development companies to become places where results, quality of life, and fun coexist sustainably.”

Being first and foremost an example of what it proposes to its clients, Pyxis is seeking Agile consultants wishing to evolve in a unique organizational structure relying on autonomy, initiative, and collaboration.

The Agile consultant will carry out projects at the clients’ site. He will act as a coach or ScrumMaster in organizations wishing to adopt or improve Agile development approaches.

At Pyxis, consultants are software development buffs. They are pragmatic and they like to work with people from various disciplines.

Main tasks and responsibilities

• Carry out Agile coaching or transition projects at the clients’ site
• Help organizations with the implementation of Agile development approaches
• Act as a ScrumMaster within development teams
• Coach ScrumMasters to help them familiarize with their role
• Participate in projects at Ensemble, our development center
• Remain at the leading-edge of Agile development practices and contribute to their spreading

Requirements

• Diploma in computer engineering, computer science, or business computing, or equivalent experience
• Over 7 years of experience in IT, including 2 in Agile software development
• Broad experience in traditional or Agile project management
• Mastering software development processes
• Knowledge of main development methods
• Sound knowledge of Agile values, principles, and practices such as Scrum and XP

Assets

• Knowledge of software engineering practices such as TDD and DDD
• ScrumMaster certification from the Scrum Alliance

Soft skills

• Dynamism
• Autonomy and versatility
• Team spirit
• Facilitator’s skills
• Diplomacy and ability to resolve conflicts
• Clear understanding of a company’s business needs, business processes, and organizational dynamics

Remuneration

Remuneration based on the value you bring to Pyxis.

In the spirit of International Coaching Week, I am offering 10 hours of leadership coaching (for free) to a leader in need. Since “96 percent* of clients report they would use coaching again to achieve personal and professional life goals such as work-life balance, increased self-esteem, productivity and much more“, this is an opportunity for you to try a coach.

In honor of annual International Coaching Week (ICW), Feb. 7-13, ICF Chapters and coaches everywhere will take time to recognize professional coaching and what it offers by hosting various coaching events within their local communities. This includes pro-bono coaching services, such as free coaching sessions, educational lectures and workshops for the public.

“This week is the perfect time for coaches to get out in their communities and show people what coaching is and how it can benefit their lives,” said ICF President and Master Certified Coach Giovanna D’Alessio. “Coaching is proven to help people attain their personal and professional goals and those who use it can expect to see a solid return on their investment.”

If you ever wondered if a coach can help you achieve better results faster, I invite you to send me an email (martin [at] analytical-mind.com) with the following information:

• Description of the professional objective you are hoping to achieve.
• Why you think a leadership coach could help you achieve your objective?
• Why YOU should be selected?

I am donating 10 x 1-hour sessions to one leader in need to help you achieve your goal. The sessions will take place over the phone (skype) at the rate of 1 session per week. You have until Saturday February 13th to submit your profile and I will select the coachee on February 15th. I am confident you will enjoy the experience.

*2009 ICF Global Coaching Client Study.

In their recently released study “Agile Development: Mainstream Adoption Has Changed Agility“, Forrester reports that “35% of respondents stated that Agile most closely reflects their development process”. The report is based on Forrester’s/Dr. Dobbs Global Developer Technographics Survey, Q3, 2009, which surveyed 1298 application development professionals.

In the past few years, Agile processes have not only gained increasing adoption levels; they have also rapidly joined the mainstream of development approaches. And while more organizations are adapting to Agile conventions, Agile is also adapting to the workplace. Perhaps the clearest sign of the mainstreaming of Agile is the abandonment of orthodoxy: Teams are puzzling out the mix of methodologies and combining them to fit within their organizational realities, blending Agile and non-Agile techniques and practices to create a hybrid methodology that fits larger organizations. Other changes, such as new team dynamics and the redefinition of roles such as the business analyst, show the genuine force behind Agile adoption. It’s time for software development professionals to stop sitting on the fence where Agile is concerned. According to those who have successfully adopted Agile, the benefits are well worth the effort, and with the recent dramatic increase in Agile adoption, the probability of working in or with an Agile team has increased for everyone.

The report looks at Agile Adoption from the following perspectives:

• Agile Adoption Goes Mainstream
• Teams Are Changing To Support Agility
• What it means
• Agile Becomes Mainstream — But Not Without Some Changes

And then offers a recommendation to get the most out of the transition: “App Dev Professionals Should Blend Agile Methods To Meet Their Own Needs“.

Agile methods encourage more-collaborative development than do traditional approaches, and many developers who have shied away from formal development methods in the past — believing them to be the province of “management” — have embraced Agile as a “formal” development process.

The report shows that of the various methodology, Scrum leads as the most adopted Agile methodology with 11% of the organizations reported using this approchach.  Other Agile approaches used are: Agile Modeling (6%), Feature-driven development – FDD (3.8%), Test-driven development – TDD (3.4%), eXtreme Programming – XP (2.9%), Lean development (2.1%), Microsoft Solutions Framework – MSF for Agile (1.8%), Agile Data Method (1.6%), Adaptive Software Development – ASD (1.3%), Six Sigma (0.9%), Crystal (0.3%), Behavior-driven development – BDD (0.2%) and Dynamic Systems Development Method – DSDM (0.2%).

When it comes to selecting an Agile methodology, Scrum is the overwhelming favorite – claiming it is simple, practical, and popular.

The report shows that when organizations select a development approach, they do so in the context of their organizational priorities and characteristics. Each organization carefully selects the approach that will best address their weaknesses rather than implementing Agile methodology for its own sake. In addition, participants considered the engineering processes to be critical to the success of the transition.

Teams do not usually implement all of these techniques simultaneously; even in the most mature Agile adoptions, teams pick the techniques that work best for them (…) This variation in the adoption of Agile components indicates that teams are more concerned with making sure they are working well together and producing high-quality software than with changing their software engineering process.

Perhaps the most important aspect of Agile’s entrance into the mainstream is the way that teams pragmatically mix methodologies. Instead of sticking strictly to a particular Agile orthodoxy, teams cherry-pick Agile methods, often including non-Agile techniques in the mix as well

Overall, the report clearly shows that organizations who have transitioned to an Agile approach use a pragmatic strategy and adapt the methodology that are best suited for their environment and context. While only 27% stick to a particular methodology, 63% mix different methodologies or mix Agile with non-Agile methodologies.

In conclusion

Agile adoption is a reality. Organizations across all industries are increasingly adopting Agile principles, and software engineers and other project team members are picking up Agile techniques. While historically, management has owned “process,” the adoption of Agile methods has pushed ownership into the hands of team members — many of whom have traditionally been skeptical of process and methodology. Broad Agile adoption requires careful consideration; a strong Agile adoption strategy should include:

• A support plan. Adopting Agile practices is not a trivial change; it requires support and time to become effective. The use of external coaches, training materials, and internal support groups can greatly increase the speed and success of adoption.
• Flexible adoption models. One size does not fit all.
• A focus on team empowerment. It sounds really easy, but this is about more than just saying that teams are empowered.
• A tool strategy. A single team in one location working alongside a customer may be able to work without any electronic tools, but as organizations scale and teams become more distributed and part of much larger releases, Agile methods benefit greatly from tools.

Related Research Documents

• “From Agile Development To Agile Engagement” - May 6, 2009
• “Ensure Success For Agile Using Four Simple Steps” - March 24, 2009
• “Make Agile Lean To Boost Business Impact” - December 17, 2008