Appendix 1
Agile Project Management
Having an adaptable process of development that can respond to rapidly changing economic conditions is one way organizations can compete effectively. Agile project management enhances the ability of teams and organizations to react to these changes. Traditional approaches to project management often entail following a set plan, and if there is any divergence from this plan the project manager is often expected to correct this or seek approval for any revised plan. Agile approaches, however, recognize that goals will inevitably change and that achieving value for the client should be the most important consideration.
It is often the case that during a development process the requirements will have substantially changed. The longer the time interval from requirements gathering to delivery, the more likely the client will indicate that what has been developed does not meet their needs. Also, it is unlikely that many of the requirements will still be considered important after a few months or even at the start of development. During workshops with clients, it often occurs that every participant volunteers at least one requirement, but if after outlining this list there is a voting opportunity for the priority of these requirements, it is likely many of them are not voted for at all. It is not only that the client team needs to see a prototype, often to understand what they truly want, but it is also likely that any software developed may change the business processes to such an extent as to make the original requirements even more obsolete.
One way to reduce the risk of development being detached from what is actually required by the client is to provide a more frequent feedback to the client including what is being developed. Agile project management accelerates the feedback cycle and actively involves the customer in the prioritization of the requirements and design of the product. Delivering a product after 12 or 18 months runs the risk that the business needs have changed or that the client team realizes on viewing the software it is not what they actually want. It is better to have a set regular feedback cycle continually prioritizing the most valuable functionality and delivering some thread of working software for the client to comment on. Producing a tangible software or product at regular intervals and having a continual communication cycle, involving the client, is at the heart of agile methods.
Traditional planning is typically based on the concept of delivering a project within a set budget within a set schedule. The agile philosophy encompasses delivering high value products or software as rapidly as possible. Ensuring that this delivery benefits from regular feedback enhances this value. It also ensures that within a fixed time the greatest value in terms of the functionality as prioritized by the client is delivered. The shift for both the organization and the project manager is one from delivery of a project to schedule and budget to one of delivery of the highest value within the time and other constraints set.
It is through a cycle of iterations and release, with continual working software of product developments, that trust is built, and the client can see that every release is providing increased functionality and business value.
The Paradox
Managers typically wish to know how much and how long a project will take and yet they still want to have the flexibility to respond to the business environment and embrace innovation. How can we achieve flexibility and respond to change, and at the same time follow a plan? The answer is partly in granularity, considering the capabilities at a high level of abstraction for planning and allowing development teams to define specific tasks according to the needs of the project. Agile is inherently measurable because of the clear regular cycles and internal and external measures. At a high level of abstraction, these are the regular releases defined by the needs of the project, domain and agile method, often every 90–120 days. Within these there are the iterations of 1 month or shorter cycles. And furthermore, within a day there is the daily cycle identifying what has been delivered, what are the problems and what will be done next. Within this, especially in software development, there are cycles that are achieved within even shorter periods by development teams using test-driven development and automated testing techniques.
Definitions of Success Are Part of the Problem
If we measure success in terms of achieving the original specifications, then measuring agile projects that are designed to incorporate changes in goals to achieve maximum business value to the client are bound to be problematic. What is needed is to base measures on what the client considers of value and for this to be updated continually. In agile methods, and particularly with the Scrum method, this is achieved via the product owner who is the client representative on the team. Typically, the product owner is part of the development team who represents and works closely with the client or client team to determine the most valuable capabilities and constituent features. At the start of each iteration, they will all be involved in prioritizing features from capabilities they have identified into a finer detail of functionality expected from the system. Dependent on the method, the requirements may be in the form of user stories to gain some idea of how users will use the system. It is through this process that the product owner with the development team prioritizes the most valuable stories for the coming iteration. This cycle continues so that at any given point the project has delivered the highest value functionality as defined by the client.
One advantage of reducing cycle time is that the team soon learns what is working and what is not, and can correct the development as necessary. Another advantage is that valuable working software or products are brought into use earlier, starting to contribute economic benefits, so that the project reaches the breakeven point and provides a return on investment (ROI) sooner.
What Is Agile?
Jim Highsmith (2010) outlines that being agile is not a silver bullet to solving your development or project management problems. He characterizes agile in two statements: ‘Agility is the ability to both create and respond to change in order to profit in a turbulent business environment. Agility is the ability to balance flexibility and stability’ (Highsmith, 2002).
The concept of iterative and incremental development is not new, and developers in the 1980s and 1990s were designing light methods aimed at involving the development teams and customers in closer collaboration. An alliance of these developers met in Snowbird, Utah, in February 2001 to see if there was anything in common between the various methodologies being used at the time. They agreed on an agile manifesto and values (below) supporting the manifesto. The latest updates to this can be found at www.AgileAlliance.org:
• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development.
• The sponsors, developer and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity – the art of maximizing the amount of work done – is essential.
• The best architectures, requirements and designs emerge from self-organizing teams.
• At regular intervals the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.
One of the most important aspects of agile processes is the continual reflection and adaption. Many of these values have also been adopted by agile project management approaches, especially the concepts embedded within one of the methods, scrum, including set iteration lengths termed sprints, daily stand-up meetings and reviews, as part of the process. At the end of each sprint, a tangible product is delivered and at the end of a series of sprints, a release, the current working version of the product is released to the customer for detailed review.
Lean
Lean processes are typified by reduced inventories and cycle times. There are many concepts that agile and lean have in common particularly in processes to remove waste and rework. The background to the lean movement can be seen in the Japanese manufacturing sector and also in the Six Sigma quality improvement initiatives. As with many agile methods and approaches, some of the ideas have their roots in previous practices.
Grant Rule (2011), during his guest lecture at UCL, 2010, outlined the similar concepts used in the production line techniques of building warships at the height of the Venetian empire, several hundred years prior to the concept of automotive production lines. During this period of expansion, the wooden warships were to protect the trading interest of the Venetians and due to the limited space in the Arsenale the inventory and waste was kept to a minimum and the pressure of conflicts meant that they had to reduce cycle times and release ships to protect their routes and territories as frequently as possible. This would be the equivalent of producing working software frequently to customers. There is evidence of continual improvement in the process although whether this allowed teams’ time for reflection and self-organisation is highly unlikely.
The importance of frequent releases of software is central in agile project management. This enables feedback from the client to build trust and an effective product that provides the highest value within the given time. As requirements often fit into a profile similar to Pareto analysis with only 20% of the functions used 80% of the time, delivering the highest value and functionality first will often be sufficient as many of the additional requirements may be obsolete, seldom used or need to be changed.
To take full advantage of agile methods, lean practices need to be adopted across the organization (Salloway, Beaver, and Trott, 2010). It is certainly the case that the major challenges in an organization are often the necessary cultural changes, which include the empowerment of teams, creating an environment of trust to allow the teams to determine their own approach to development.
Agile is not a panacea. There is no set recipe to follow, but there are some patterns that perhaps all projects should follow. One pattern is the idea of time-boxing every aspect of the project cycle. This is not just a defined time but as short as possible time to speed up the feedback and associated learning cycle. This includes the discussions with clients as well as the review meetings.
Two Levels of Planning
Agile techniques encourage planning at two levels of abstraction. The customer or client, represented by the product owner, usually has an initial idea for the capabilities of the product being developed. This allows for a high-level capability plan to be developed in which the capabilities can be valued. Dean Leffingwell (2007) suggests a value feature approach in which the features are assigned priority and value, giving immediately some indication of value of the project as well as an approach to track progress in terms of value at a high level.
At an iteration level, the capabilities need to be decomposed further into features and stories and be prioritized. It is this that gives the second level of tracking and can be achieved in terms of stories and size estimated in terms of story points. The team then decides how these tasks are broken down and commit to a daily level of delivery of this work.
Terminology
As with any professional practice, there is often terminology that may be difficult for the outsider to interpret. In law, Latin phases are often used, although increasingly in other areas such as medicine it appears that the language is converging to more readily understandable terms. A glossary of agile terms can be found in the Guide to Agile Practices http://guide.agilealliance.org/. Perhaps it is that sometimes those not involved in agile project management and development are somewhat perplexed by terms such as scrum and planning poker. Is this some kind of game we are playing? Well, part of the answer is yes, as the highly interactive and collaborative planning process has been termed a ‘game’ by some leading agile proponents (Cockburn, 2002). A game involving the customer at its core and delivering the highest business value within a given period.
How Does a Generic Agile Development and Project Run?
A workshop may be needed to determine the overall process and project management approach. Different agile techniques favour different approaches, and use different terms. All outline the need for prioritization of requirements. In some methods, such as feature driven development (FDD), these are known as features and are similar to other approaches that incorporate an understanding of how the user makes use of the system, such as by user stories.
User stories tend to have more clarity if developed in conjunction with the client, and time is spent on considering how they will be tested. The user stories are stored in a product backlog and the product owner or representative from the client organization determines the value of each of these and prioritizes. The development team then determines which are done within the next iteration. This is achieved by an initial planning meeting at the start of the iteration often lasting only 2 hours in which the stories are estimated by the development team as to how long these will take. The team also takes into account the priority within the product backlog as to which stories are to be developed for the next iteration, and reviews this when the iteration is completed.
Stand-up Meeting
At the start of each day, there is a stand-up meeting. The idea is to keep the meeting short to a maximum of 15 minutes and to encourage comments to be kept succinct. Here team members outline briefly what they did yesterday, what were the impediments to getting certain tasks done and what they are going to achieve today. Tasks and who is doing them are summarized on the whiteboard, and because these commitments are made to the group there tends to be motivation to achieve what individuals have outlined as their tasks. After the short meeting, any technical problem discussed will often immediately be resolved by others in the group depending on whether it is an architecture, programming or resource issue. There is regular feedback. Typically, there is a retrospective or review at the end of each iteration to outline what went well and what did not go as well, but there are also daily reviews and the stand-up meeting helps highlight problems early.
The key is a built-in system of reviews at every level.
With my research groups after their projects, I asked them, if they were allowed to repeat their projects how long would they take to achieve the same results? Most had been working nearly 6 months and agreed that to repeat their work and achieve the same results would take less than half the time. Much of this is due to the learning curve, working out how to solve a problem, deciding the valid metrics, developing effective testing procedures, designing the tests and introducing automated testing procedures wherever possible. This is the importance of speeding up the cycle time to get results quickly and learn from them. To develop code and deliver this to the client after six months without a review with the client is a recipe for disaster. They will invariably say that this is not quite what they had envisaged. The feedbacks are necessary both for team learning and also to ensure that the teams are delivering the right product for the customer and ensuring that what they deliver is of the highest value and quality. (Collins 2013)
It is important for the research groups to consider the value of the project management approaches and consider them as assisting rather than being an overhead. Techniques in testing clarify the goals and if testing is automated should accelerate the process. A more facilitative approach to project management is required both in research and development. Within the scrum method, this is supported by role of the ScrumMaster whose function is to ensure that the processes determined by the group are followed.
Estimation
In traditional approaches, managers estimate and try to establish predictable plans, and any deviation from these is seen as a problem with the development team. This approach, often combined with a waterfall sequence of requirements for gathering, design, development and testing, is often a failure. The long list of requirements that may have taken months to gather is often out of date before the design is started. This is why it is so important to build something for the client to see at regular intervals to check that what is being built meets the requirements.
Mike Cohn (2004) humorously pointed out in his book on user stories, using an analogy based on shopping, that seemingly trivial tasks often take several hours to complete. This is a way of introducing several important points about over-optimism in estimation, and not allowing enough time for technical development or thorough testing.
Of course, there are times when project managers may deliberately tell senior management what they want to hear, and agreeing to an unrealistically short schedule that the development team has no hope of achieving. With agile project management, these aspects are substantially reduced. First, the ‘death march’ scenario of no time and unrealistic planning from the project manager is avoided because the teams estimate their own work. The priorities determined by the client in fact decide what should be drawn from the product backlog and incorporated into the next iteration. This backlog can then be used as a clear measure to protect the team from unreasonable additional demands. If further tasks are added to the product backlog, it is the priority of each task that determines the urgency. As only a certain number of stories in terms of relative difficulty and duration in story points can be achieved in any iteration, there is an immediate indication of whether it may be feasible or even if it is a priority to add the new stories into the current iteration. The work is then delivered at a sustainable pace in a predictable way, and the project schedule is fine-tuned as more process and project metrics become available.
One aspect of agile is using the ‘wisdom of crowds’, tapping into the fact that groups can estimate with better predicted outcomes and faster than individual team members. Earlier approaches have used Delphi techniques based on averages and giving higher ranking to central values. In recent years, a popular technique that has been adopted by the agile community is ‘planning poker’. This allows the rapid estimation of user stories and will allow, based on these estimates, the team to plan a realistic set of user stories and constituent tasks for the current iteration. It is based on the concept that those doing the work are best placed to do the estimation. Also, as many of the benefits of estimation are quite quickly achieved, putting in more effort has decreasingly diminishing returns on accuracy.
Each of the development teams has a set of cards marked with a series of Fibonacci numbers. These are made up of the previous two numbers added, 1, 2, 3, 5, 8, 13, with the larger card not necessarily fitting in the sequence, say 100. There are other variants of this series that begin with 0 and 0.5, others based on multiples of 10 and others even with a question mark for the decisions that are difficult to initially make. The idea is that it is relatively easy to determine how long a task will take relative to another task, but this becomes more difficult as the size increases. As it becomes more difficult to differentiate between consecutive numbers as the task becomes bigger, the widening gap makes it easier to estimate (e.g., it is easier to determine whether a task will take closer to 8 or 13 units than say 11 and 13 time units).
For each user story, each member of the team estimates in story points the relative effort it is going take to develop and places their selected card with the value face down. As soon as everyone has done this, they turn their cards over to reveal their estimate at the same time. There may be consensus on the numbers say 3, 3, 3 and 5 in which case the card representing the generally agreed value is taken. However, if there is variation, a low and a high number say, those who selected these values outline their perspective, and may be in light of this there are assumptions and technical issues that the others have not considered, and there is another round of voting. If there is consensus the team can move on to the next story. Rounds without discussion, as outlined by Amr Elssamadisy (2009), allow reflection and consensus, but may have the consequence of individuals being too influenced by the perceived leaders in the group.
If the team is working with user stories, then they need to estimate the relative size of these stories at the beginning of the iteration as a team. Mike Cohn (2004) points out this can be done in story points, which can be relative to the team’s own working practices and experience. However, the true measure of working becomes apparent from the team’s own measures of velocity, how many of their story points they are completing within a week or iteration. This measure can then be used as yesterday’s weather, to provide improved estimates of the rate of work, and better baseline as the measures for completion of user stories and their constituent tasks to accomplish these (Cohn, 2006) (Fig. A-1).
The team uses its own measures such as velocity and ‘burndown’ charts that show the remaining stories to be developed. As capabilities and features become increasingly clear and stable from management, and the team acquires a sustainable development rate, the test data will support the figures to show when the project will be complete. The test data including percentage passing acceptance tests, the amount of code changes (churn) and the defect rates are going to be of particular importance in software development and will be used in conjunction to establishing realistic estimates for the overall completion of the project.
Technical Debt
During the planning process of the iteration, it is essential to consider the quality of the product and ensure attributes such as scalability and security. These may not always be outlined by the customer or the product owner, but to avoid technical debt and the code being difficult to maintain and extend these issues need to be addressed. It is important in the planning process that architectural issues are dealt with. One way to help achieve this is to consider not just the user functionality or user stories but also tasks that have to be achieved in order to keep the architecture aspects required. This can be achieved through a dependency mapping as outlined by Brown et al. (2010).
Defining the Architecture of the System
The architecture of a system describes its overall structure, components, interfaces and behaviour. Definitions vary but often include the perspectives or views of the structure (Bass, Clements, and Kazman, 2003). One area that is emphasized by one instance of the generic unified process (UP) and rational unified process (RUP) is the concept of requirements. In this case, it is often user functionality via user cases and subsets of scenarios that shows what functionality a user or actor requires of the system.
Some agile methods such as extreme programming (XP) favour the use of understanding of the system in an exploratory way, via development of code, improving the design without affecting the behaviour, i.e., refactoring. So, for instance, developing a security check in a banking system would necessitate understanding the structure and coding this feature would verify and quickly establish an architecture, if a model is not already available.
Unless the architectural issues are addressed, there will be inconsistencies in performance as the system is scaled. These defects will require ever more refactoring in order to avoid design problems. Therefore, some consideration of the design is necessary to avoid later problems. This process of consideration of the planning of the architecture is termed architectural runway, allowing for a smooth transition and rework and to avoid technical debt. Planning ahead, the architecture can be allocated on a planned process as advocated in agile architecture provisioning (Brown et al., 2010). Here, architectural elements that are necessary for quality attributes (non-functional) such as security are allocated to the iteration backlog and provisioned within this period to carefully outline both functional and non-functional requirements, which are so important in determining scaling and performance factors. The rationale for architecture decisions should be recorded and can be incorporated within iteration planning as shown in Fig. A.2. An alternative but similar approach would be to include design spikes at the last responsible moment (LRM) to ensure the flexibility of the architecture. When there is a choice of architecture, this can then allow for a different approach to estimation of value. Instead of using a static cost–benefit analysis, which is normally based on static estimates and architecture, an investigation of alternative options, and their relative future changes can be investigated and interpreted via real-options analysis (Bahsoon and Emmerich, 2004). This approach, which was originally developed in the financial markets, is increasingly used to determine the dynamically changing nature of viable options in areas such as provisioning of resources as in cloud provisioning within the IT sector (Collins, 2011). The nature of engineering is changing and increasingly ‘composing systems from open source, commercial, and proprietary components’ (Bosch, 2011), and in agile environments where the focus is on early exploration the ‘selection and trade-off decisions’ should be captured including the rationale that will help to understand why the product is better and why it is being built (CMMI, 2010). Pareto optimality and how this can be applied to balancing requirements as well as trade-off decisions for goals within project and programme management is a current area of research within the Faculty of Engineering Sciences at UCL.
Sharing Knowledge and Reflection
Research teams are well known for their synergy. Jeff Sutherland (2005) outlines this enthusiasm in agile projects and Dean Leffingwell (2007) outlines these concepts including how teams can create new points of view and resolve problems through dialogue. Within workshops, particularly in scientific research and innovation, it is useful to allow dialogue, not having to defend your idea, and further time to explore possibilities. This is refreshing, as often the stress is on discussions to resolve issues within a specified time period, i.e., time-boxing. Leffingwell also points out that in the spirit of scrum, amongst its many attributes including commitment and autonomy, ‘leaders provide creative chaos’. This is precisely the concept that Sir Paul Nurse, during the BBC David Dimbleby lecture in 2012, was conveying when he was outlining how to create a collaborative environment for researchers to excel at the future Crick Institute.
Earned Value
With traditional project management progress is based on tracking the intermediate tasks, such as production of the requirements document and the design artefacts, which can be achieved without a demonstrable or working product. Allowing measures to be unrelated to working products can give a false sense of progress.
With agile the focus is on whether the software works, whether it is what the client intended or it is of value. This is done through continual feedback via releases to demonstrate and allow feedback to improve the product and value.
To track progress towards goals in technology and IT projects where there is emergent design, this needs to be achieved at a higher level of abstraction. In research projects, a clear idea of initial goals or exploration of concepts still needs careful planning; otherwise, it will be unlikely that funding will be granted.
There may be an exploratory phase equivalent to a feasibility study in which one or two workshops are planned to outline strategy or develop architecture to develop the technology. This initial phase will have clear goals and should have clear rationale for those invited. As this is bounded by time and consultants and facilitator costs, the budget can be easily ascertained. This is likely if parties agree to an exploratory design phase, then a development phase.
During the initial phase, in order to define the scope or boundaries of the research or development, it will cover the goals that will in many instances then be broken down further to capabilities according to the type of innovation or development project.
Capabilities, a term often used in military projects, determine what is required without determining how this will be achieved. If this is a software development project, these may be subdivided into features and stories. Sometimes, the term epic in agile development is often used for a larger aggregation of features.
User stories represent what the client or product owner requires. These requirements are often written as a short outline on a card and then decomposed further to tasks. The product owner discusses with the team the prioritization and order of development with regard to development issues including software architecture.
The use of earned value management (EVM) is well developed in certain sectors of project management including the construction industry and is being increasingly mandated in defence projects in the UK. This can be applied to agile and software development. The essence of agile development is this shared collaborative communication between client and development team, ensuring value for the client and a motivated development team. Although management may be used to using earned value measures, this needs to be developed for the agile process so that the development team does not see this as an overhead, and both groups can work collaboratively together. One criticism, however, of EVM is that this does not give any indication of the quality. One benefit of adapting this approach in conjunction with agile project management is that processes that improve quality, such as refactoring, are often incorporated into agile development practices.
Figure A-2 Architectural elements in iteration planning (Collins, 2011). Based on Brown, N., Nord, R., & Ipek, O. November/December 2010. Enabling agility through architecture. CrossTalk.
If EVM is required by senior management or through the governance process of the project, one approach is to create a reporting on two levels; one for the capability tracking and one based on stories as shown in Fig. A-3, which can be generated with minimal overhead as part of the planning process.
In agile development, there is continual prioritization at the iteration level, which is designed to be the same duration. Earned value can be applied to these user stories, but there is a subtle difference in application. The reconciliation seems difficult, as the user stories are continually under prioritization according to the client as to what they consider the most valuable user story. These stories are stored on a backlog for selection during each iteration, and the team estimates how large they are in terms of story points.
The stories according to prioritization are selected for the iteration by the client on their perceived value. If the table is prefaced with the functional value or business value, the order of priority can be much clearer, although the highest priority should be at the top of the backlog, and the development process then pulls the next set of stories and set of tasks.
Figure A-3 Earned value derived from stories completed at the end of an iteration. Story points are an estimate of the relative time that the development team thinks the work will take to complete.
As can be seen from the figure, the earned value can be derived from the amount of completed work. For the first story, with the highest priority on the list and business value, the story was completed and therefore earned all the points.
Although the story took slightly longer than expected (i.e., 120 hours), it was complete and therefore earned the full earned value of 100. When the story was not complete, it was allocated zero. From these figures, schedule performance index (SPI) and cost performance index (CPI) can be ascertained to give an indication of progress (Collins, 2006).
Some have argued that velocity is more important. This is not the same as earned value. Velocity is the rate at which a team works and is a useful internal measure as are burndown charts, which show the remaining work and can act as a motivator for the team.
However, while it can be seen that earned value is valid within an iteration, there are different interpretations of how this could work at higher levels of abstraction and the value to project management at a project level. Craig Larman (2004) suggests the use of estimates in terms of budgets. This is most easily achieved in terms of hours of work. He also outlines the concept of re-calculating the planned work for each iteration, and the baseline is updated as more information arises. For the tasks to earn value, it is prudent to only consider these when fully completed. However, assessing this needs careful consideration as the development of each story is usually considered finished when all tests including integration and acceptance tests are completed.
It can be seen that using a simple spreadsheet as a by-product of the team’s estimates, an EV system can be used as an indicator of progress in terms of business value.
Emerging Trends for Agile Project Management
Agile project management provides the opportunity to rapidly deliver software, services and products to satisfy the demands of ever-changing customer needs. The need to rapidly develop software and digital processes using agile methods has become increasingly important in order to compete in digital markets, indeed for many organizations it is becoming vital for their survival.
One of the trends is the provision of a continuous delivery of software and services. This approach uses concepts from agile methods and lean manufacturing, incorporating frequent releases and automation wherever possible. Regular feedback from customers ensures the quality of deployments and what is delivered actually meets customer needs.
Continuous delivery can be far more effective with the integration of development teams with delivery teams. Considerable further gains can be made if software, service and project delivery can be integrated with other business processes. The challenge for managers is to ensure that all relevant processes are optimized, providing an opportunity to reduce the cost and complexity of processes across the organization.
Agile projects are becoming increasingly data-driven to ensure that they are meeting the goals of the project and aligned to what the customer requires. The progress of projects is now often tracked real time, based on working software or solutions and feedback from customers. It is the governance and compliance aspects of project management that are increasing in importance in agile approaches, ensuring that process improvement, business goals and regulations are being met.
The following sections examine some of the major trends and challenges: understanding customer needs, use of data and visualizations, achieving collaboration across teams and providing teams with autonomy so they can select whichever lean and agile methods they consider add value to achieve the business strategy.
Delivering Enterprise Agility
Although there has been considerable success in software delivery using agile methods, the challenge now, as Rigby et al. (2016) outline in their Harvard Business Review article, Embracing Agility, is that many organizations need to capitalize on agility across the organization. In an increasingly competitive market, it is necessary to gain the advantages of adaptability provided by agile processes. Evidence for the economic benefit has been shown by surveys extending back to 2007 by the Center for Information Systems Research (CISR) at MIT Sloan, indicating that agile firms can increase profits by 37% (Weill, 2007). In addition, Peter Weill and Stephanie Woerner (2016) have shown there is a significant increase in revenue and profit margins for organizations that embrace digitalization and understand their customers better.
Research published by McKinsey showed that many organizations are using agile project management to deliver goods and services with greater efficiency (Comella-Dorda et al., 2016). This also brings further challenges from ‘always on’ customers who expect continual availability and reliability. To take advantage of new market opportunities and adapt dynamically requires not just adaptability within delivery pipelines but across the enterprise. As Ross, Weill and Robertson explain in their book, Creating a Foundation for Business Execution, to keep up with the changing business environment, it is also necessary to have flexibility within the digital processes and enterprise architecture (Ross et al., 2006). It is with agile project management that organizations can take advantage of these new digital business opportunities and accelerate time to market.
Understanding Business Value and Customer Value
Forming strategic partnerships and looking for ideas outside the organization is one way to accelerate innovation. This can be aided by more agile approaches allowing teams to decide how to structure these conversations and change how teams share information at the start of projects, via workshops or alternatively hackathons.1 These approaches provide an insight into the processes and value streams of different groups. For a customer, this may give further insight in their value stream, from their request to fulfilment of their order.
Agile processes are increasingly integrated with different perspectives so that the teams better understand the customer’s journey. Customer needs are increasingly incorporated at an earlier stage, not just after the engineering solution, as we now often see, integrated with agile processes the mapping of the customer experience (CX).2 Processes need careful design and consideration, whether they are providing value for the business or customer, or both. For example, it is of limited value if, an app helping a customer locate a store selling a product they want (via geo-location on their smart phone), if this system is not integrated into the supply chain and the store takes weeks to order the product. The whole supply chain needs to be considered. Mark Schwartz highlights in his book, The Art of Business Value, that managers need to ensure that the meaning of value is considered from the perspective of the business, partners and what the customer values (Swartz, 2016).
Integration of Processes
Many organizations are using agile project management to meet customer needs and improve the integration of their processes. Siemens is an example of a global organization who are not just focused on agile software development but use agile project management to focus on business value, reducing lead times and release times, and getting feedback from their customers. These processes allow flexibility, so that if something does not meet needs of the business, or the customer, it can be adapted. Siemens also integrate their application lifecycle management (ALM) to their product lifecycle management (PLM), for example, with the manufacture of embedded systems. The sharing of information across these processes makes production of these embedded systems seamless and lowers production costs. It also allows requirements to be traced more easily. Integrating PLM software and agile software development practices enhances information systems and is seen as a necessity to help deliver the corporate strategy. This not only allows the consolidation of different systems but also enables global engineering groups to collaborate and launch products as one team, reducing time to market.
Siemens also achieve rapid feedback within their prototyping, for example, within their autonomous robot manufacturing systems, part of Siemens agile manufacturing systems (SiAMS). At Siemens Corporate Technology Centre Princeton campus they are building mechanical spider-like robots to work in hard to access areas, in manufacturing or surface preparation. They are optimizing them to work together using algorithms that mimic the knowledge sharing and collaboration of human agile teams (Siemens, 2016). In fact, these robots are being designed for other attributes you may expect from small agile empowered teams, self-improvement and the ability to share tasks autonomously.
To fully capitalize on agile processes, enterprises do not need just high performing individual agile projects but take advantage of agile methods across the organization so that the entire supply chain is examined and optimized.
Continuous Delivery
Problems typically arise when teams work in isolation. Integrating the teams responsible for development and operations (DevOps) allows for a continuous delivery pipeline. Releases of small functionality can be made available in smaller increments to get feedback from customers and reduce risk. If there is a problem with a release, the fact that only a small amount of software is affected means that this can be quickly rectified. This is further enhanced by what is known as blue-green deployment, having two identical production streams ‘blue’ and ‘green’. Blue will be live and have the production traffic, while green will be in the final stage of testing. When green is fully tested and ready, it is switched so that it is live, eliminating downtime. Additionally, if a problem did occur then there could be an immediate roll back to the blue version further reducing risk.
Cultural Shift
DevOps can be considered as a cultural shift, encapsulating the philosophy of cross-functional teams, with both development and operations collaborating effectively to improve the flow and delivery of products. To be fully effective this needs to involve other related business units as well. This can be easier to achieve in organizations that fully embrace agile methods than have a bimodal solution, where traditional and agile modes coexist. However, having predictable outcomes and moving to a more exploratory state, if managed effectively, can facilitate the transition from traditional processes to digital processes while reducing risk of a ‘big-bang’, an all at once, transition.
A natural progression in DevOps is to ensure that all relevant aspects of the business and development are involved in the process, including the quality assurance and the project management office (PMO). The PMO supports the delivery of projects in an organization, monitors review of the delivery of projects and provides decision support.
The conditions for project success, as indicated by the Association of Project Management survey (APM, 2015) includes, ‘effective governance, the project needs to have clear reporting lines and effective communication between all parties’. Governance frameworks ensure goals and standards are being followed and allow organizations to make investment decisions. When integrating agile practices, the role of the PMO needs to be considered to ensure a suitable fit to the strategy. However, it can be a challenge for the PMO to have the necessary and up-to-date skills to provide advice for continuous delivery teams. Lloyds Bank has recognized that having a technical capability within the PMO is vital, and Sanjeev Sharma, CTO, DevOps Technical Sales and Adoption, IBM, explained at the DevOps Summit London that the banks embed an expert from the DevOps team within the PMO to facilitate this process (Sharma, 2016).
For agile focused organizations, where the priority is to favour digital transformation, there is often a need for a specialized PMO. This structure can be thought of as analogous to the structure of programmes, with PMOs having different specialities supporting relevant groups of projects. This can enhance governance and compliance in global organizations as with IBM, which adopted this kind of structure with specialized PMOs for each of its product lines. However, as Guy Barlow points out in his Oracle blog, the real-time sharing of information becomes even more critical if the role of the PMO is decentralized (Barlow, 2014).
DevOps when integrated with other business units responsible for understanding the customer needs, quality assurance (QA) and the PMO can considerably accelerate the successful deployment of software and services. Collaboration across all teams related to delivery and deployment, ensures a reliable and resilient delivery pipeline of software being continually adapted to customer needs.
Data-Driven
These changes throw up new challenges so that software and enterprises are increasingly becoming data-driven.
There is a shift in agile project management towards projects being driven by real-time data. It is through this continual experimentation using empirical methods and agile and lean approaches, based on the scientific method, that projects can be managed in fast moving digital environments. Although the planning may be reduced within exploratory and fast moving digital projects, feedback as to whether the project is meeting the goals of customers and the business is often enhanced. Organizations are realizing that the project management aspects of agile processes are ideally suited for the necessary governance, compliance and tracking that the strategy and customer needs have been met.
NASA’s regular feedback from development of their customer-centric software from their nightly builds also helps them keep track. The priority for NASA’s teams is to place emphasis on tracking the code and their goals through daily feedback from their customers rather than upfront planning (Trimble and Webster, 2013). Their measure of progress is increasingly working code. This analytics is becoming more important in continuous delivery where problems can be identified in real time.
It is the ability to rapidly get feedback from customers that is changing project planning to an emphasis on data driven processes establishing whether we are meeting customer goals. This may be in the form of A/B testing, where a portion of the web-traffic is being driven to one group of customers to see if there is an improvement in sales, and if so then the whole traffic is switched to the new version. Humble, Molesky and O’Reilly outline in their book, Lean Enterprise, that this method using data from this method is used by technology organizations, such as Amazon and Microsoft, to see if features in their projects will even add value before it is built (Humble et al., 2015).
As organizations move to continuous delivery, this throws up further challenges. Managers need to provide an environment of psychological safety and trust, in order that their teams are willing to experiment and continually improve, via processes, such as improvement katas,3 as Humble et al. (2015) outline. Skelton and O’Dell (2016) point out that it is the cultural issues, supporting teams via manageable workloads and ensuring software architecture aligned to team structure that is important. Joe McKendrick, quoting the CTO of Amazon, Werner Vogel, outlines that Amazon makes it ‘easy for developers to “push-button” deploy their application’ and release updates for customers, in effect, achieving deployments at a rate equivalent to one every second (McKendrick, 2015). If organizations are to replicate the success of Google and Amazon, and be able to adapt their processes within seconds and meet the ever-changing needs of customers, agile and lean methods incorporating rapid feedback and real-time data are required.
Shared Vision
To avoid silos of ideas and to ensure projects take into account all necessary viewpoints, workshops are typically held. Facilitated workshops at the start of projects are integral to many agile approaches, such as the dynamic systems development method (DSDM, 2008). Rally Software, now part of CA Technologies, also integrates its workshops with release planning. Their approach which termed, ‘agile big room planning’ is designed to ensure business units across the organization are brought together and collaborate from the beginning so that everyone is on the same page. As part of this process, teams co-locate to participate in the planning process. This ensures communication with the business, that the project is aligned with the business goals and that the entire organization is engaged and has an input to delivery of the product or service being developed. Seagate, a data storage organization, adopted this method to identify bottlenecks in its processes and improve predictability, achieving a regular cadence for release planning.
Research by Dingsøyr et al. (2016) has shown that it is important in agile software development that teams have a shared mental model (Dingsøyr, 2016). Having this shared vision is equally important as we scale projects and integrate other units across the organization. This can be enhanced by visualization tools, which can be adopted for DevOps, to bring the tasks for both streams of operation and delivery into better coordination.
Kanban is a lean concept that is becoming increasingly adopted to improve visibility of progress across teams, reduce waste and ensure that work in progress (WIP) is kept to a minimum. Tasks are written on coloured sticky notes, which are placed on a whiteboard, called a Kanban board, to track the progress across stages of development. Jaguar Land Rover uses Kanban software to reduce lead time and improve flow. The software also allows continuous feedback, allowing different engineering teams to add comments and improve the decisions. The visibility of progress is further enhanced, by showing the status of each activity, including assessment. When activities are assessed either, ‘assessed OK’, or ‘not OK’ is posted. The latter is linked within the software to different categories, why the assessment has not met the required standard and real-time discussions as to further work that needs to be completed. Key decisions are also recorded including architecture hard points, such as wheelbase or position of seat with respect to other features. The discussions and diagrams illustrating any problem arising can also be shown at the same time, so that a seamless resolution of issues can be achieved.
Systematic, an international software company has successfully deployed and scaled agile methods for its software development. Agile processes are applied across the entire organization, including their management systems, improving visibility and reducing the document workload. Their teams have been an early adopter of visualization and automation and within their continuous delivery pipelines for clients. This has resulted in systematic being one of the first organizations to be accredited the Capability Maturity Model Integration (CMMI) level 5 showing the high level of repeatability within its processes. This efficiency allows a focus on business critical areas and customer satisfaction.
Ensuring Everyone Contributes
Agile teams are increasingly using open source messaging apps, such as Slack, for sharing their communications. This has partly given rise to enterprise agile development, increasingly providing a more interactive environment and often allowing developers to discuss and share ideas in a social way. Communication tools incorporating a social dimension can ensure that a wider range of ideas is explored. Executives are starting to recognize that harnessing the creativity of external stakeholders as well as that of their internal teams can improve knowledge sharing and be a catalyst for innovation. Harrysson and co-researchers showed that social technologies can invigorate collaboration and help develop strategic insights (Harrysson et al., 2016).
One of the challenges of scaling agile project management is often the boundary between teams, particularly with external stakeholders (Strode, 2012). Here, the communication strategy needs to be carefully managed. It is often a project manager needed in this role to coordinate these teams, although there are various approaches to scaling and designating roles based on scrum, including the scaled agile framework outlined by Richard Knaster and Dean Leffingwell (Knaster and Leffingwell, 2016). It is however recognized in methods that a representative with detailed knowledge of the business (or product owner) is central to the prioritization of goals, and should, as Mike Cohn emphasizes, ideally come from the business domain to fully appreciate and communicate the business priorities (Cohn, 2014).
Aligning teams to the architecture of the system can enhance communications between development teams. Aligning teams to traditional functions can cause a considerable amount of additional work with hand-offs, and waiting for teams to deliver work to other teams in the process. It is important to design teams according to the communication structures using Conway’s Law: which outlines that organizations are ‘constrained to produce designs which are a copies of communication structures of these organisations’. Sam Newman outlines the need for flexible architectures and that these provide opportunities for modular architectures to reflect the structures of teams to improve efficiency of communications (Newman, 2014).
Nvidia uses agile methods throughout its development processes. For example, in the development of its graphics processors, it arranges real-time interaction with customers to enhance their designs. It not only uses real-time interaction and feedback from customers during development of the products, but also uses its products in the design of the building to facilitate interaction and collaboration. Deborah Shoquist, Nvidia’s vice-president for operations was quoted in the International New York Times outlining the use of their ‘computational power and technology to model their new building in Silicon Valley’ (Markoff, 2016). Nvidia is using its highly interactive rendering software in conjunction with their graphics processors to design its offices. This allows teams of architects, designers and engineers to work in collaboration with state-of-the-art virtual reality (VR). These design teams can make changes to problems that only become apparent when they interact with the 3D visual model, avoiding costly alterations later in the project.
Dingsøyr et al. (2016) have shown the validity of agile principles and clear goals. They have also established that it is important that decisions are discussed. Even discussions where there is disagreement for the processes or technical approach required have been shown to improve teamwork. In addition, research published in the IEEE journal Software by Van Heeach et al. (2014) indicates that it is also important to record key decisions. This, for example, has been shown to be effective in the design of their decision-centric architecture reviews, where not only the decision chosen but also alternatives are documented, the pros and cons for the chosen solution, as well as envisaged future issues which may occur. Those disagreeing with a decision need to have their viewpoints heard, if the teams are going to value each other’s opinions and accelerate their learning and performance.
Agile Does Not Mean an End to Planning
Unfortunately, there is still the misconception among some senior executives that agile does not need a process and that agile is tantamount to anarchy. Rigby et al. highlighted this impression within their Harvard Business Review article (Rigby et al., 2016). However, to truly take advantage of agile project management and processes, the reverse is true: clear lines of communication, roles and responsibilities are necessary. There is also need for a consensus-driven approach that is equitable, inclusive and transparent.
A documented architecture certainly provides clarity but the strategy and business processes still need to be considered in terms of the operating environment and changes in regulation. The throwaway aside by the CEO of the middleware software company Software AG ‘if you are agile you don’t need a strategy’ reported by David Cassidy PCPro, August 2016, suggests that if you are an agile organization and adopt their enterprise software suite a strategy is not required (Cassidy, 2016). It is certainly unlikely that modular software aimed at workflow can fully understand your customer and business needs, security and data storage issues. If anything agile should enable the technical teams to focus in depth on architecture and development, and therefore free up more time for management to focus on the strategy of their organization. Agile development will also allow teams to rapidly hone-in on the desired needs of customers and test whether the strategy is correct and evolve this to their needs. With the adoption of continuous delivery, you cannot have a monolithic architecture but need a modular architecture, such as microservices: small domain-focused services. A modular architecture certainly helps but still needs careful consideration, if this is going to be resilient and scalable.
Delivering Value Not Bureaucracy
Agile teams are typically provided autonomy, so that the team decides which agile practices and tools they should adopt and make them more productive as a team. Spotify reduces bureaucracy by encouraging its teams to prioritize agile principles over specific practices as well as encourage the teams, called ‘squads’, to select processes they consider add value. So, for example, a squad may omit burndown charts if they consider this does not add to their effectiveness. Spotify also ensures that members treat each other as equals: every aspect is designed to reflect this. Even the label for role of ScrumMaster, which can be construed as someone in-charge, rather than mastery of a process, is termed an ‘agile coach’, as explained by Henrik Kniberg, to emphasize that this role should be a servant leader (Kniberg, 2014). Typically agile teams are empowered and determine what is important to deliver the functionality. Brian Bergstein illustrates, with an example from MIT, that managers also need to recognize that it is the high level of collaboration, and sometimes a willingness to break the rules, which triggers innovations (Bergstein, 2016). Throughout the agile processes managers and teams need to ensure that bureaucracy is kept to a minimum.
An observation from history provides a reminder of the importance of autonomy. In writing about the industrial production within the Arsenale, the shipyard in Venice, during the 17th century, Joanne Ferraro outlines that the government of the time ‘was successful in co-opting workers’ support by permitting them considerable self-governance’ (Ferraro, 2012). This is one aspect that can be applied today in project management in adopting agile processes, providing trust for teams to make their own decisions, but balancing this with direction and clarity of goals. Providing the appropriate balance of autonomy and governance is key, if we are to support disparate groups to contribute and deliver as one team.
Summary
In agile development, it is often about trends emerging rather than making guesses about the future. Walker Royce wrote about the indicators for converging on the solution and the indicators for value and progress (Royce, 2011). The way to gain credibility is through working with the client on a joint understanding of a problem, how to measure progress and when to converge on the solution. This creates the real value, not only to the business, but to the self-worth of the team.
Agile project management is about achieving value collaboratively for the project and client team, and the organizations concerned. This is not just about the bottom line but achieving something at work, feeling valued and sharing knowledge with colleagues to achieve that next breakthrough. This is the true value of agile for individuals, the team and the organization.
At UCL at the front of projects are different types of workshops, examples of agile patterns that bring the right mix of researchers and project managers and support staff to solve problems. These vary from the Town Hall meetings where the challenge and opportunity is outlined to more detailed workshops allowing discussions and dialogue. The agile project management and development approaches favour the time-boxed approach to discussions, and estimation and often adding more time does not necessarily give a better outcome. It is these workshops which are often the driver for new innovations and approaches to development. The use of a trained facilitator is often vital with large programmes. It is important that staff with different technical approaches can outline their view. It may be that the idea is rejected in favour of an alternative idea discussed at the meeting. The key is fair process and that the team are deciding the direction. This is the essence of agile allowing the client to work with the development team collaboratively to decide capabilities they wish to develop and prioritizing the functionality and user stories, or in the case of research the investigations. (Collins, Graham, UCL Research and Consulting projects (2003–2013), 2013)
It is self-evident that if one member of a team suggests a technical solution, another may disagree and point out an alternative, and why in certain circumstances it is a better resolution.
Developing real-time modelling agile approaches with colleagues had a significant impact on communication and indirectly in one project resolving political issues by the clear focus on the technical problem rather than individuals. During a consulting assignment my colleagues in a small consulting group were asked to outline our object and business modelling approach. We had been asked for a solution and found on arrival at the client site already strained working relationships with a development project that had been on-going for over a year. It was agreed to use our agile modelling approach to clarify the goals of the project and be clear on the direction. It was clear things were not going well, the project manager complained that he didn’t know what kind of project he was working on, as it wasn’t properly defined by the programme director. He didn’t know whether it was a business transformation project or a business improvement project. The client really wanted the project management problems to be resolved and not upset the director of the consulting firm under contract. It was a mess and an expensive mess with technical teams having worked for a considerable time. Without a real-time modelling solution and experience in forming a unified team this impasse would not have been resolved.
Getting buy-in wasn’t just a problem of clarifying the goals but getting support by under- standing each of the stakeholder’s goals, communicating the direction over a short iteration with a clear product and defined time and engaging all stakeholders already working on the project.
The leader must be seen by others not to be gaining personally. In the case of the programme that had to be put back on track, it was imperative to listen to other parties and support each of their objectives so that the programme could move forward. Self-interest other than wishing for a successful outcome was not in the cards. Likewise, the leaders in agile project management must lead by trusting their team and allow their team to deliver the project in a self-determined way. Self-organizing teams and allowing them to report on progress are areas that the leader must embrace in agile project management. Much of what has been written on agile has been on what the teams do and how they track their progress. The burndown chart, keeping progress visible and keeping tasks visible on the wall are for the team and the team leader. Keeping key tasks and communications visible, this is the ‘whitebox’ of progress reporting. It is the leader in agile project management who needs to understand this iterative process and be a resource provider, to remove all impediments to the team, who must trust his or her team to deliver in the technical approach they consider best. It is this beyond anything that defines the change to a leadership culture in agile.
The challenge in agile project management is not prescriptive plans and practices to follow, but to populate the project-planning process with appropriate patterns that are effective and add real value. For the time being, the challenge must be to balance the planning so that you can achieve the flexibility to deliver increasingly complex projects and rapidly add new developments to enterprises and research establishments.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.