One of the things UX has going for it is that when teams get a little taste of user experience work, they typically want more. What we do is attractive because it brings value both to the user and to those who engage us, it is emotionally compelling and aesthetically delightful when done well, and it is about human stories with which people can empathize. But relying entirely on the addiction of client teams to user experience can also lead to a service model that only provides incremental benefits. To advance user experience in your organization you should find the areas of maximum impact and implement a strategy to get there.
A Case Study: Creating a User-Centered Culture
The approach I having been using in the IT organization was recently summarized in
Lund (2010). The situation is similar to the one you might face when starting UX from scratch in a company or large organization. The IT organization is made up of several thousand employees and while the majority of the employees are based in and around the Seattle area, significant numbers are based in India, China, and elsewhere around the world. Even in the Seattle area the IT staff is distributed widely across various buildings. IT produces hundreds of applications each year to enable the company to create its products, to support employee development and
productivity, and to support operations. It also produces applications used around the world by our many partners, and applications that support our customers as they buy, use, and receive support for our products. The role of IT in helping create a relationship between the company and its customers and partners has been growing even more critical in recent years. When I joined IT it was because there were champions in senior management who believed that user experience is critical to help the company to achieve its goals.
While the user experience population across the company is sizable, within IT the population consists of a relatively small number of professionals primarily distributed across my group and a group supporting internal applications (e.g., Human Resources). If the proportion of UX within IT was based on the demand of a company with a similar size (
Lund, 1994b) or based on a proportion similar to the product groups, it would be many times larger. Many in IT, as with most large corporations, fell into the trap of assuming that employees are captive users and will have to use applications whether they like them or not. They neglected to realize that poorly conceived and designed applications hurt the productivity and effectiveness of employees, and therefore of the business, even if they do use them properly. Research and experience at many companies demonstrates that employees may be “captive” in the sense that if told to complete a form they will, but that does not mean the form will be completed accurately and used in the way senior management imagines. We continually run across situations where people work around the system to check off the list of requirements without actually having to experience the pain of the application. They get their jobs done in spite of the applications, not because of them. As a result, decisions further upstream are based on inaccurate information and therefore ultimately hurt the business.
The approach that is wired into many of us in this situation is to grow by making the return on investment (ROI) case, and demonstrate either the bottom-line value that UX brings or at the very least how UX satisfies the strategic goals of the business and of management (
Karat & Lund, 2005;
Lund, 2007;
Lund, 1997a). Even management occasionally asks for the ROI justification, although when provided it does not seem to be as persuasive as people would hope.
Dan Rosenberg (2004) eloquently pointed out that these arguments alone in the best of times do not seem to be particularly effective in actually driving UX growth. Often it is far more important to either have a strong advocate in senior management or to leverage the direct value that a team experiences as they receive the benefits of UX work when driving growth. We have been fortunate in having advocates within senior management ranks who seeded user experience and nourished its start. Growth in these circumstances, however, is often slow. On the ground, an organization has to decide whether a vacancy is going to be used to bring in another developer, tester, or project manager or to use the vacancy for a user experience person whose role they are only beginning to understand. Even if growth was not an issue, it would take years to grow to satisfy the need within IT; in the recent economic climate that rate of growth would be unrealistic. These challenges are faced by nearly any large organization or company where UX is just getting a foothold.
The approach that I have been taking is to step back and instead of relying on the incremental approach, to think about what it means to transform an entire organization of thousands of people and to grow a user-centered, design-thinking culture across it, and to do it with a handful of designers and researchers.
The Approach
I should be clear on the scope we are attempting to address. Within an organization as large as IT or an entire company there are various groups within the organization that will have their own cultures. Across the entire company there is often a corporate culture that is the collected set of values, traditions, and other elements that characterize the company, which senior management often tries to steer. I am interested in influencing the IT organizational culture — “the specific collection of values and norms that are shared by people and groups in an organization and that control the way they interact with each other and with stakeholders outside the organization” (
Hill & Jones, 2001). In essence what I want to propagate throughout the organization is a system with a common vision that is shared across the organization, a language that all can use to talk about it, a reinforcement structure that motivates people to take the appropriate actions to improve the experiences created and that shapes how individuals interact, and enhance the way they think about and interact with the artifacts they create. Furthermore, the goal is to design this system around a kind of virtuous cycle of self-reinforcing activity whose impact grows as it operates across the organization.
In raising the design-thinking IQ of an entire organization the quality of the hundreds of experiences created improves faster than if you have to rely on the small UX population and the projects directly supported. It scales the efforts of UX up, and indeed allows them to scale indefinitely as the greater organization grows. It transcends both geographic and discipline boundaries. It should build into the organization shared belief in the value around UX and drive the kind of demand for expertise and value that may lead to bringing in more designers and researchers (e.g., full-time staff, contractors, or vendors).
I also want to be clear that an implication of this approach is that non-UX people are being given more UX skills. I recognize up front that this is a frightening or at least disturbing prospect for many, and growing a UX empire would certainly be better for my personal career. If non-UX people grow their skills will they conclude that UX is no longer needed? Will they do a terrible job and ruin the “brand” of UX? Both have occurred in the past and will occur again. On the other hand, it is not clear whether it has been because of the training or in spite of it. Developers, project managers, and others will drive user interfaces without UX. Whether or not UX people are working on a project, an effective user interface and a great experience is a collaborative effort. If approached by everyone based on a foundation of sound design thinking the design will be better.
My experience is that the more people know about user experience, their own impact on the experiences improves and they come to respect the expertise of professionals even more; that respect often leads them to be evangelists for hiring more
user experience people and getting them involved. These evangelists in turn do a better job of collaboration with the user experience people with whom they work (often challenging them, but in the process driving a better result). Furthermore, they often continue to grow their own skills and in some cases become professional UX people, bringing their own diverse experiences to enrich the field and projects. In short, I have found bringing more people into the UX tent has the benefit of being the engine that leads to growth. The alternative, where UX is owned and driven only by the UX team, would take longer and potentially be less impactful than when large numbers of people with higher levels of design thinking are engaged in the process.
Project managers should be better project managers and when they work with user experience people, they should be able to facilitate the creation of even better experiences as experience producers. Developers should be better developers as they build the support of great experiences into their code. Testers should complement user feedback by not just testing the code for traditional bugs but by empathizing with users as they ensure users can achieve their goals through the targeted, contextualized scenarios.
Engine of Culture Change
The approach I have taken is to define a model of culture change, a virtuous cycle that as it operates and the cycle iterates the culture evolves. It is illustrated in
Figure 5.6.
At the heart is the vision for the experiences we want to create. This vision provides the language for the common direction and the business rationale for that direction. In the approach we are discussing here, it includes the core ideas that drive business value and adoption. It includes the experience attributes we want to deliver, the design experiences we want to create, and the themes we are trying to push through our designs. For the IT organization, at the core is delivering compelling, useful, and easy-to-use experiences that satisfy the brand promise.
Organizations are not necessarily motivated by models and supporting research. They are motivated by sticky ideas — by stories and images that grab their attention. As a result we are developing ways to grab their attention around the vision. We have been exploring how to further refine this vision in the form of concepts that capture the relevant elements. For the division we are in, the core concept has been growing the relationships between the user (whether customer or partner) and the business.
Deliver and Inspire
There was pressure in the first year my IT team existed to avoid supporting specific projects and to concentrate on creating standards for user interfaces. My experience has shown, however, that the best standards and guidelines are grounded in successful design rather than pulled out of the air. We have persisted, therefore, in pushing to ensure that the user experience teams are focused on the most important projects within the organization — and to identify those projects that are strategically important and where the user experience is in the critical path to success — and to get involved in those projects. In addition, we continually search for the difficult business problems facing the organization, and identify problems within that set that are centered on the experience. We work on breakthroughs for those problems based on design exploration and user research.
Our goal to move into new areas is to invest in setting the direction, the vision of where the design should be several years out (even before there is technology to support it), and to ground that vision on the user’s needs and desires (the usefulness that drives the satisfaction). That vision is where we attempt to drive innovation in what the experience could be for users. Against that vision we work with teams to create road maps of how we will get there, and then the bulk of the team’s work is the beginning-to-end involvement that delivers on the road map.
When great design is delivered it raises the visibility of design thinking across the organization. Senior executives get excited about the vision and even about the solutions to their problems. They get excited about the fact that their users are engaged in the process. The executives become evangelists for great user experiences, and become the sponsors for implementing the new ways of talking and thinking about the experience artifacts and how experience is woven into the design process and the teams that engage in it. The great design also becomes a source of the reusable assets and best practices, and by their strategic weight the design lends credibility to those assets. Finally, there is great design that teams without user experience can copy when appropriate.
Equip
An underutilized area where the user experience impact can be extended is through the architecture and the reuse it enables. The architecture we build our solutions on assumes Microsoft and partner platforms. On the platforms, there are a variety of data sources required for the business. In the middleware layer, systems correspond to business capabilities. Above the middleware layer, capabilities are combined to support user applications. The applications are then rendered to support
an expanding ecosystem of devices as users move from device to device, and to support groups of users collaborating as part of various activities. This is where the usefulness is delivered, and contextualizing the experience through end-to-end user scenarios that are supported in the design is how we intend to deliver satisfaction. Integrating user experience into the architectures used by the teams is one way to insert the experience into the DNA of the organization.
When I started with my current team an architecture review meeting was held. I noticed that the presentation of the architecture only had a small annotation that there were user interface issues. We volunteered to create the poster that would show off the diagram, and I volunteered to co-author the architecture document. By the time we finished, more than half the diagram consisted of layers that connected the underlying systems with the users in context. There is a lot of power in being the author of the meeting notes, and there is a lot of power in being responsible for communicating the architectural ideas visually. Just through the involvement in the communication process we were able to change the language and the mental models of the teams.
The goal of the architecture we created is to create user interface building blocks that can be combined in various ways as new solutions are created. By using the same building blocks across applications the users will associate the family of solutions with the same brand and use the branding as a cue that triggers scripts that will help them know how to use new solutions. It should set up expectations about ease of use and usefulness. In addition, the goal is to assemble blocks into groups and groups into higher order groups based on common needs. The primitives should reflect sound usability principles, and the higher level design patterns should be both useful and easy to use; skinning with various brand identities should change the voice and suggest the support for families of scenarios, but should not be the basic ways people use various controls to get things done. This model placed our work on common guidelines and reusable design patterns at the heart of the strategy that was being used to turn the organization into a more efficient development team.
As part of this strategy we have taken several steps. I became a senior UX sponsor for a cross-MS icon database and a cross-UX community design pattern wiki project. We have led or participated in IT standards efforts around measuring user satisfaction, implementing accessibility and internationalization, and we are planning standards for usability measurements and branding. A major project of the team has been to build a UX Pattern Framework for implementing UIs so they can be easily rebranded and support the pattern architecture. We have housed this evolving set of patterns in a developer library and implemented it for several projects. We are working with another design team to create a library of reusable design assets and page templates.
The core idea is that IT should build from the conventions developed across the company and the partner ecosystem. In doing so, we leverage existing knowledge to enhance ease of use, add value to that ecosystem, and also serve as a customer that can provide user feedback into the ecosystem to drive innovation and improvement. We can also generate knowledge that can be transferred to our customers and
partners about new ways they may also get value from the ecosystem. While executing this strategy I have been pulled in to talk to customers about how we leverage the ecosystem. The message is that user experience is integral to the success of that application development process.
The building blocks can be put together in ways that still result in bad user interfaces. To help with that, we have defined a set of common user requirements that should apply to all solutions throughout IT. These include accessibility requirements, for example, as well as requirements for supporting global localization and to support responsiveness in the user interface. If we succeed in creating truly common requirements then project managers, developers, and testers will become familiar with them, making them part of the standard practice, and they will only rarely need to be referenced. Education will then only be required around updates. Furthermore, to help obtain broad buy-in across the organization, the change management process is intended to incorporate requests from the organization to ensure that the practical day-to-day needs of UI guidance are reflected in the document; anyone can file bugs against the document to drive improvements within it. This process has the side benefit that the entire engineering organization is able to develop ownership of the evolving pattern of reusable assets available for designs created by the organization. An internal survey showed that the common requirements and the human interface guidelines are viewed as two of the most anticipated useful documents created by the UX team.
Having the right architecture is only part of the challenge. The blocks need to be assembled in a way that results in satisfying and compelling experiences. We began by defining the waterfall process illustrated in
Figure 5.7. The starting point was the development process that had already been defined for our IT organization, a process that is typical of one with a planning phase, a requirements phase, a design phase, a development phase (including testing), and then deployment and maintenance. The process is somewhat complicated by the organizational stakeholders. For most of the projects within IT, certainly the most strategic ones, there is a business organization that sponsors the initiative and drives the business plan. Within IT, there is an organization whose responsibility it is to translate the business goals into business requirements. When I started my career back at AT&T Bell Laboratories, this was the equivalent of the systems engineering organization. The business requirements are then taken by engineering and turned into specifications and coded. At AT&T Bell Laboratories, this was the development organization.
We overlaid onto the existing process the set of user experience activities that should take place within each of the phases if teams have full user experience support. The core concept is user-centered design. Understand the users, translate that into design, test, improve, and iterate, and do it from beginning to end. This is enhanced with the idea of engineered creative design thinking, baking in a process of broad exploration and then prioritization, focus, and renewed exploration for the next phase.
For teams that are encountering user experience for the first time one of the questions we are asked is What do you do? Another question is What do I do with it? which can be read as Why should I care? In describing how user experience
fits within the existing development process, we showed how our deliverables are influenced by the deliverables on other teams, and how our deliverables influence subsequent deliverables and make them better. The goal is to describe more clearly how the quality of the user interface development and the success of the project is enhanced by engaging user experience throughout the project, and how design thinking and user understanding can influence every phase of the project from uncovering new business opportunities, to development, and to validating that the project was successful (e.g., through the satisfaction and other metrics described previously showing the impact of the work). In particular, we have been able to show where user experience activities improve the usefulness of the result and where they address ease of use. We use this framework to lay out a very clear description for who is responsible for what and to inform project planning.
The goal is not just to focus on what the UX team does for projects, but rather to provide a user-centered perspective that any team can use. When UX is involved, the process should work better and achieve an even more spectacular result, but every project should benefit from design thinking. There certainly will be some teams with full UX support, some with support that will come from engaging contractors or applying a smaller UX team for specific tasks, and some teams without any UX support.
One particularly effective approach to defining a set of tools that raises design thinking across the organization has been to drive a scenario focus into the development process. It began with a quality initiative within the organization. We defined quality in terms of process changes we wanted to make to raise quality. These included building it right (with a focus on things like release management and test automation), but also an effort around building the right thing. Building the right thing focused on clearly identifying the users, defining the scenarios that drive business and user value, and then ensuring the scenarios are delivered. We used this as an opportunity to take many of the concepts and tools built around user-centered design and integrate them into the scenario-based development process.
One issue that arose early was the diverse ways in which people define scenarios. I remember quite clearly sitting in the evening in front of the TV surrounded by books, articles, and materials from one of my sibling UX teams (where they had evolved an approach that pragmatically worked for their team) and coming to the conclusion that instead of trying to debate various types of scenarios, I would just define the terms in the way we would use them across IT. The goal was to define terms like scenario and use cases in a way that would allow the creation of a development system that would support creating usable design. From the definitions I created templates that would be used to transform our requirements and specifications documents. From those definitions it was then possible to create various tools such as scenario-based bug classification. This bug classification allowed us to build usability into what was otherwise a purely engineering-focused process.
We then began driving scenarios into the larger organization with the support of the general manager. Teams were required to create quality plans that explicitly laid out how they would implement scenarios. Over time the goal was to raise their
levels in a maturity model that showed how they were using scenarios for user-centered design. I integrated scenarios into the release management process, and that allowed us to track progress publically in the organization. We built scenarios into the code management tool the developers used, and that allowed bugs to be filed against the scenarios to improve them. We put together training around the process and launched training and mentoring with the teams as well as drove it within the projects we supported directly. The person on my team who had been driving this effort (Peter) moved to report to my boss directly so he could drive the overall quality process. That continued to raise the emphasis on the scenarios work across the organization.
Serendipitously, the training organization was developing a scenarios-based emphasis. We fed what we had been learning and developing to them, and they integrated scenarios training with a user-centered design course for non-UX people. They began rolling that training out more broadly. Peter, in turn, moved to the IT version of a training and engineering best practices organization, and he took the course and contextualized it for IT using our scenarios approach. He then took on the charge of driving the training through all of IT. As a result, thousands of people are being trained in user-centered design, and the demand for design and user research has been blossoming.
Personas and Scenarios
We have built and are continuing to expand a library of personas that teams can use. While there are differing views about personas, for a situation like ours where they stimulate empathy through their stories, they serve as an effective anchor to help teams begin to think like the intended users. They also help teams focus on specific users rather than on “all users.”
We have defined a specific form of scenario, templates for how to structure the scenario, and guidance on things to do and things to avoid, and we have assembled best practice examples. The scenarios are prioritized, and the goal is to adopt something like the Kano model and
Pruitt and Adlin’s (2006) persona-based prioritization scheme to help teams think through the process of prioritization based on the needs and desires of the personas.
A new step we are introducing is to standardize the expectation that early sketching and rapid prototyping will occur as the requirements and scenarios are defined. The key for integrating UX and user-centered design into the development process is that this early prototype provides a kind of common model and language that all the stakeholders in the process can agree on, and serves as a representation of how the scenarios might be implemented. It is understood that it is not a final design, and indeed is expected to change as the design is more fully explored and tested with users. It is a starting point for testing with users, it demonstrates the feasibility of implementation, and it puts a reasonable size around the design effort (number of screens, number of controls and design assets that will be required, etc.). It also helps to identify where further and more detailed explorations will be required.
The scenarios are then turned into use cases by the project managers and documented in the specifications. The use cases serve as the implementation requirements for the specific application being built. Test cases are then built from the use cases, but testers also use the personas and the scenarios to ensure they are staying focused on the goals the users have and not just on the software itself. We are implementing a bug severity and prioritization scheme that introduces the user experience, the scenarios, as an equal player against the more technical bugs that can occur; and to build it into the way test, development, and project management think it is being built into the tool that is used to report, triage, and track all the bugs. Reports from the bug reporting tool are sent up the management chain who will then review experience bugs based on the targeted scenarios and require accountability around delivering the experience. Finally, traceability is defined and tracked from the test cases back through the use case and to the scenarios; this provides some assurance that what is being built is what users are saying they want and need.
The scenarios are the backbone of the user-centered process, and are an attempt to anchor the process at the beginning. When UX or market research is involved the research is better controlled — and will certainly have higher reliability, generalizability, and validity — but we believe that even without experienced researchers teams will be better off if they listen to users and adapt designs based on what they hear. Then our goal can be to improve those skills.
The scenarios, however, need to be turned into design. Teams without UX will produce designs in the course of project management and development. We can further extend the user-centered model, however, if we introduce some form of interim prototyping and validation between the requirements and the specifications. To do this, the plan is to leverage Microsoft’s SketchFlow tool in the Expression Suite, and build the reusable controls and design pattern components (the building blocks) into it. Teams will be able to put the building blocks together to try different implementations to support the scenarios. Based on the experience either they will generate better use cases or be able to expose them to users to get feedback on whether they effectively support the implementation of the scenarios and then use that feedback to generate the use cases. Another tool that we have begun to leverage is one that allows prototypes to be exposed to user communities who then provide feedback based on the previews. The feedback should inform changes to the prototype. Furthermore, these teams should be able to draw from the libraries of design patterns and in some cases copy the best practices from the visible, highly strategic projects that the user experience team has been supporting directly.
The result is validation of the intent of the design with users up front and user-centered focus in requirements, design exploration and confirmation with users that what is being built seems to deliver the usefulness being targeted with the scenarios (and the context the scenarios reflect), user “surrogate” testing carried out by testers who are educated with heuristics and grounded in the validated personas and scenarios, and a feedback loop from the users once the application is deployed.
The feedback loop ensures that the problems will be pulled into the planning for the next release and teams can iteratively (albeit incrementally) improve the experiences they create.
Training
The most critical step in creating tools that transform the way people work is to create tools that are usable, useful, and easier to use than existing alternatives. But just making them is not enough. We need to support their use and grow design thinking (
Brown, 2008) across the organization. That means we need to train and mentor. To do that, we have put together a curriculum for our IT organization that raises the design-thinking IQ of the entire organization. In addition we are engaging senior management support to grow the UX expertise of a subset of their project managers, developers, and testers. This combines courses my team created, those developed by the other IT UX team, and courses offered corporately. The curriculum is illustrated in
Figure 5.8.
We have structured the curriculum into levels with basic courses, intermediate courses, and extended courses. Initially we thought we might emulate the Six Sigma Black Belt concept, but then felt that we did not want to imply that if you had a black belt you had everything you needed to be a fully skilled user experience professional. Instead, we decided to present the curriculum as teaching the ability to do very specific tasks. The tasks would enable teams without UX to do a better job in what they design, and would enable those taking the classes to partner with UX more effectively and for UX to scale their impact ad design direction beyond the simple number of UX people on a project. Since the designers and researchers need to deliver inspiring design as the core of their day job, part of the goal is to drive toward as many self-service courses as possible. We also want to leverage the existing training that the engineering training organization is offering, and complement it with the training that we create, which is specific to our local organization.
There is an introductory class that teaches design thinking, the vision, general principles of great design (
Lund, 1997b), and the process and how to apply it. It also provides an introduction to the available reusable assets. There is a class in prototyping with wireframes, and a class in simple validation with users that includes several sessions and hands-on labs. There is a class on scenarios, and one on implementing the reusable assets and customizing CSS. There are also hands-on workshops that we are developing with the quality organization around resolving user experience problems. We are also working with the corporate training organization to see if we can leverage their registration process to track who is participating, and to bake the training into the objectives of individual project managers, developers, and testers who share responsibility for the quality of user interfaces.
What we are enabling are UI specialists within each of the disciplines. The UI project manager is someone who is responsible for the end-to-end scenarios that deliver usefulness and user satisfaction — not as the person who owns
developing the experience, but as a producer who engages others to design and develop the experience and facilitates its creation. The UX project manager works well for many teams, ensuring teams without direct design and research support have a better experience, and improving the effectiveness of teams with design and research support. The UX champion makes sure the key activities that need to happen do, and does so by making sure that someone is responsible for each activity and reporting status up through management. UI developers will be trained in the new Expression-based development tools as well as the asset libraries and principles of how to put the blocks together in the most effective manner. UI testers will be trained in how to best use personas and scenarios to frame the questions they ask during testing, in the bug classification scheme, and in design heuristics.
Motivate
Motivation within an organization begins with defining metrics that will be tracked and that matter to the organization. The model within the vision suggests that satisfaction is at the heart of what we want to measure. One step is to evangelize a consistent measurement and to drive broad adoption. We have helped drive a standard for this metric. It is often collected by sampling the user population as they use the released application, often by using user panels, so the value is in motivating release over release improvement. The goal is to build the metric into the system in a way that ensures it is exposed in the standard dashboards used throughout the management chain and is discussed during strategic and tactical review sessions.
The next step is to increase the visibility of the metric across the management chain. That alone is usually enough to stimulate action. No team wants to have a yellow or even red box around user satisfaction next to their project. There is an opportunity, therefore, to drive more activity around satisfaction by building user satisfaction into these goals. For more senior management, the goals would include either achieving specific target levels or achieving some percent increase year over year. For people closer to the frontline, the goals might include either target levels for usefulness and ease of use or operationally equivalent performance-oriented goals.
A satisfaction metric alone does not provide enough guidance about how to make improvements. It needs to be supplemented by more diagnostic tools. Typically the satisfaction metric is collected as part of a larger study, and if we can define an appropriate framework that deconstructs the experience along the lines of the vision model into components that are meaningful to the user, we can identify the degree to which each impacts overall satisfaction and therefore business value. Furthermore, if we can drive a consistent set of user feedback tools across the organization based on value, ease of use, and other factors, not only will managers have information about what is working and not working for users, but also they will be able to compare application experiences and make better decisions about the relative importance of various experiences and where to invest in improvements.
One class of tools would capture what groups of users do (and how long it takes them to do it), while keeping individuals anonymous unless they are part of a predefined user panel and have given appropriate permissions. Another would similarly give users the ability to rate ease of use, usefulness, and satisfaction with various aspects of their experience, and to provide verbatim feedback about what they were attempting to do, their needs and wishes, and so on. If this feedback could be tied to the behavioral tracking we would have a rich set of data for planning improvements. Some of this feedback could be channeled through a tool within management dashboards or as a desktop gadget. Feedback from real users expressed in their own voice creates empathy and would motivate change. Our current challenge is that there are many tools being used, but we still need to get to a more consistent approach because none of the tools fully supports the integrated view based on the acceptance model.
My general manager, however, has noted that while clearly user experience is critical for project success, metrics such as satisfaction are so far downstream and are influenced by so many factors that it is hard to associate the user experience work with the success. He asks, therefore, that we identify leading indicators of that value and success. If we can identify key performance indicators that are highly correlated with success, then we can track those. The ease of use and usefulness subjective and objective measures are part of those key performance indicators, but potentially so are the presence or absence of certain artifacts within a project (like scenarios), and other forms of quality assessments such as sign-off on the quality of deliverables, the level of user experience bugs, and other indicators. We have a long list of potential indicators, and one of the challenges is narrowing these down to the handful that it makes sense for management to track and that are effective predictors of project success. We are also defining a framework of metrics that can be tracked through instrumental applications based on the targeted scenarios of use. If we can define a consistent framework, reports can be generated that allow projects to be compared and managed, even if different tools are used to collect the data.
Leadership
The biggest threat to effective user experience I have found is UX being treated as a service organization, as a team that produces usability studies and icons on demand. To drive culture change, we have felt we needed to lead. Some of that leadership comes with having a vision, some with inspiring, and some with equipping and motivating. But some of the leadership comes with identifying the strategic questions for the organization, and the subset that can be answered most effectively through user-centered insights. We then define the big bet initiatives we want to undertake that will move the needle for the organization’s success in a big way, which is equivalent to the impact other teams have when they propose and deliver major new software initiatives. These big bets might involve putting in place a major user panel, a global ethnographic study, or working through a major experience problem whose
solution will impact architecture such as implementing identity in a cloud computing environment.
Currently we are in the process of rethinking our corporate Web site and how to drive more consistency and integration across it. The challenge has been that it has evolved as a loose confederation of sub-sites. To rethink and reshape it we are working with the corporate marketing and design group to establish the vision, but to really drive the vision through the site our current objectives are to inspire the sub-sites we support with a vision of what their experiences could be if they were aligned with the common vision.
A different area of leadership is about how we can scale trained design and research support across IT, even though there are limits to the full-time staff IT is willing to hire. In the past this has been done by hiring contractors and vendors as they are needed. Often they are paid two to three times the cost of a full-time staff and have had varying levels of success. Whether they are hired directly or through us, there is a considerable delay in bringing them on board (both in finding the talent, and working through the contractual process). This reduces team agility and ends up focusing UX work on the least impactful parts of the process (the production parts of design). The approach I am taking defines and implements a staff augmentation model.
The staff augmentation model involves projecting what the demand will be more broadly across IT, and then establishing a contract with one or two vendors to source the skills needed. I have projected a prototypical mix of deliverables, and vendors have established an initial estimate of prices based on that mix. More important, however, is the creation of a project management office (PMO). This office will handle recruiting talent based on my criteria, ensuring training and compliance with our standards and guidelines, and will define and report key performance indicators. It will have the ability to expand support indefinitely. Over time, I will be able to drive further improvements in quality, and as the volume of support increases vendor prices will drop. Since the PMO will report to me, I will be able to ensure that work well beyond my own organization’s charter fits within our culture change strategy, will be able to leverage the PMO to evangelize and grow UX work across IT, and hopefully will be able to use the network of contacts to persuade teams to repurpose vacancies among their engineering staff to bring on full-time UX people.
We are in the process of implementing this strategy, so the jury is partially out on how well it will work. It is encouraging to note that the feedback received when we began defining and implementing this strategy two years ago was that almost no one thought about user experience in most areas of MS IT, and now many acknowledge that everyone talks about it. Requests for user experience support have skyrocketed, and user experience is getting visibility up and down the management chain.
5.1. Five Management Dimensions in Managing a Usability Design Team
Mark S. Hoffman, CPE
Director, Store Operations, Technology, and Ergonomics, Business Strategy Group-Retail, Dover, OH
Managing a usability design team that is responsible for developing user experience solutions for commercial business is challenging, in today’s engineering and manufacturing environments with fast product life cycles, of new technologies in consumer IT products on future product experience expectations, and the trend of large companies to share design and development process with lower cost off-shore resources. As the pace of technology growth and adaption continues to increase, the role of usability developers, human factors practitioners, and ergonomists are challenged to improve the effectiveness of usability product design process, and market knowledge about end users in the markets served.
During my management tenure, I had the opportunity to develop and manage a human factors/ergonomics organization that was chartered with the responsibility of usability design for products and systems used in global commercial environments. During this time, the company changed ownership several times, frequent changes in the senior management team.
The design and development of products and systems for commercial industrial markets is unique in the IT industry, because new solutions have to accommodate legacy hardware and/or software technologies and provide improvements in usability and systems performance. Unlike personal consumer technology where radical innovations in design occur in very short life cycles and the changes drive demand, product innovation in commercial industry tends to be subtle and stepwise. To the casual user, the usability innovation is less obvious. Because of this, many times in product development, usability design often takes a secondary seat to changes in product appearance.
During my tenure in management, the human factors/ergonomics usability team grew from a small group to over thirty FTEs with several university research partnerships, and then was subjected to a dramatic decrease in support with equivalent reductions in headcount as the team was forced to shift from a hybrid organization, engineering development and professional services, to a professional services organization. I will share lessons learned in defining a management and growth strategy building an HF/E team and managing the team from its beginning through the migration from a development organization to a profit center. The following five dimensions provide the manager with insight defining the strategic plan and the environment for the team’s success.
Understand the company culture, past and present. There are often subtle differences in group cultures and departments, senior staff, and corporate management can have different group cultures in large companies. As a manager, you will need to determine the appropriate strategies for working with these different groups, especially when their preconceived perceptions about the usability design team inhibit the team’s effectiveness. Many human resource departments follow the Hay Point methodology for compensation and particularly in organizations that are high horizontally differentiated with matrix management. It is important to know if the organizations’ perceptions about the user experience design teams’ role and attitudes from the viewpoint of senior management embrace a strategy that limits your career and the team’s future growth so you can plan accordingly.
The frequent disconnect between manpower, resources, and budget needed to support product development and management control of overhead costs continues to conflict with the budgeting process of R&D organizations. HF/E was one of several engineering functions in an engineering services organization. As with many vertically differentiated organizations, there was minimal expected headcount growth from the outset of the establishment of the team, annual budgets were controlled by the engineering development teams, and advanced development engineering was assigned to the corporate organization. I quickly identified the need for additional headcount to adequately support the five different system developments and
the manufacturing operation. My foremost challenge was to determine alternative methods for getting additional support without significant team growth in the headcount supported by the engineering budget. Recognizing that there were significant differences between corporate and divisional organizational cultures, I found it useful to use a macroergonomic approach similar to that described in Hendricks & Kleiner (2001) to characterize the organizational cultures within different teams and identify decision autonomies and latitudes within departments, divisions, and corporate cultures within the company over my career in management.
Know the senior management team that owns the manpower budget and what priorities are taken into consideration during the budgeting process. One rule of thumb I learned quickly was that there never are sufficient project resources allocated for usability design and development. This can be a difficult lesson for many practitioners moving into a management position to accept, especially when they see their peers in engineering development rarely have to address this problem. Practitioners quickly learn to balance workload and design quality within schedule constraints and then find additional resources to cover critical shortfalls. Finding additional monies to supplement the budget is challenging.
Many managers choose to outsource work to contractors with lower overhead costs to fill short-term gaps. Managers should map the frequency of these gaps to determine when it is an appropriate time to increase department headcount and identify a strategy for securing additional sustainable funding given the long-term cycles of annual budget cuts and shrinkage in headcount.
I used graduate interns to supplement the team and developed customer consulting services for users of our products and systems. This required developing close alliances with the direct sales force, external customers, and industry trade associations. I implemented a management strategy in resource planning that expanded practitioner experience and expertise in industry business and operations practices through project assignments. This provided the platform for the interns to learn business practices so they could support marketing and sales teams. Recognize that transforming a development usability engineer into an industry consultant requires three to five years because of the nuances and differences between industries and global practices, and job performance objectives.
Plan for short- and long-term growth, attrition, and retention. The staff recruiting challenge is to balance diversity in expertise with the skill sets used in solution development. Most organizations tend to choose the safer option, recruiting candidates with similar training and expertise rather than intentionally recruiting broader expertise with diverse experience. I chose to develop the team by intentionally recruiting a broad range of expertise that spanned across the diverse fields of human factors, ergonomics, systems engineering, and industrial engineering. Managing diverse expertise and training is both challenging and rewarding. Resource scheduling over the long term can be difficult because the demand for knowledge and expertise from different areas of specialization vary depending upon the product development cycle for engineering, as well as project deliverables for customer consulting engagements. The key benefit of a diverse team of specialists is that each practitioner has the opportunity to broaden their understanding of usability design and strengthen individual design and evaluation skills. Care should be taken to identify and recruit individuals who are willing to be flexible and learn new knowledge and welcome the opportunity to acquire new skills that are outside of their preferred areas of expertise.
Support team and individual professional growth and recognition, both internally and outside of the company. As the team manager, you have the challenge to identify individual strengths, weaknesses, and preferences for work assignments for each team member. You need to provide the opportunity for their participation in industry and professional associations, which requires funding and time commitments that further complicate the budgeting process. Planning for internal and external recognition for all team members is critical to the long-term stability of the organization. Recognition
through design patents, internal and external publications, professional certification, and support for participation in industry research consortiums provides a good platform for their individual growth.
Adapt your management style to reduce disconnects between changes in company growth objectives and department workload demands. As the manager, you have fiscal, functional, intellectual property, and personnel management tasks that can conflict with unexpected changes in company direction for senior staff. Recognize your personal tolerance for risk taking and supporting unpopular policies and positions, and develop your interpersonal communication and interaction styles knowing your limitations, preferences, and strategies for growing the team. This needs to be done by keeping cognizant of your personal career goals and objectives (see
Figure 5.9).
In summary, setting a management strategy and adapting the strategy to drive changes within the organization is an ongoing process, especially in today’s volatile markets. The five dimensions in
Figure 5.9 provide a platform to anchor the manager’s strategic plan.
5.2. Lessons Learned in Managing a UX Consultancy
Simo Säde, PhD
CEO, Etnoteam Finland, Helsinki, Finland
I am sharing these practical-level thoughts and experiences from the point of view of a manager of an independent user experience consulting company. Our UX team has grown over the years, and most of the development of our activities has happened without a preexisting framework or guiding company policies. Below, I will describe some of the challenges we have faced in organizing our work and in developing our business. We have succeeded in some of the issues below and learned some of them the hard way. With many of these, we just need to keep on working.
The consultant firm environment is very different compared to an in-house UX team in a large corporation. The projects we work on are typically shorter than in-house development projects. Forecasting the project situation in the future is far more difficult and the time span of a reliable forecast is short. The importance of sales is more directly related to the team’s everyday life.
Challenge of sizing the team to meet the market need
In consulting, usability and user experience projects are often short, as we typically participate only in a certain part of the overall customer project. Also, our sales cycle is short. It is common that the projects do not fall into our calendar nicely and evenly. There may be high peaks in workload and periods of quietness after that. The projects in a consultancy firm are often gained through bidding competition, and it may happen that there are several bids in at the same time. Winning or losing several bids at the same time may mean either a serious lack of resources or serious oversupply of them. Furthermore, the won projects are often confirmed just days before kickoff, so the reaction time is short.
High variation in the level of workload means problems in sizing the team. One solution is to have a core team supplemented with consultants under hour-based contracts. Normally this flexible resource consists of junior specialists, such as students, who work part time. This model may cause problems with communication and project quality. It requires more hands-on management tasks from the senior staff and more hours used for familiarization in the projects, which causes inefficiency. There is also the risk of not meeting the normal quality standards. Once an hour-based consultant has been in the company for some time and has gained experience, these problems eventually disappear. For the manager, this situation may cause an additional challenge. It may be difficult to keep the hour-based consultants happy, if they would like to be nominated to permanent full-time positions but growth does not allow it.
Another better, but more fundamental, solution is aiming at a large, widely spread clientele that can provide an even workload. Growing with dependence on one industry or on one or two main customers is risky in this sense. A recession hitting the industrial field in question or some sudden strategic change in the main customer may mean that outside consultants are no longer needed.
From individual to shared competencies
When a small team of experts begins to grow — outside a well-regulated environment of a large corporation — it is easy to realize too late how important it is to document the way the work is done and the best practices. When a small number of colleagues work together they know what each of them are doing and they develop the methods, processes, and skills together. With a bit of growth, when there begins to be several parallel projects and separate project teams, it is important to document and communicate. Mentoring a younger colleague informally, or sharing the lessons learned in corridors and on coffee breaks works for some time, but soon alternative, competing ways of doing the same thing start to appear, causing confusion, inefficiency, and uncertainty.
Written procedures and processes, checklists, and instructions not only make it easier for the newest team member to get in and start working effectively, but they also allow the whole team be efficient. As a very basic example, all the document templates needed in the team
must be consistent and easily available. But even if they are, it is common practice to create a document by modifying an existing one. This way various, slightly different versions start to bloom and lead to an inconsistent, error-prone way of working.
Having lessons-learned sessions may serve also as team building effort. When team members discuss the successes and failures in a finished project, it grows the feeling of being in the same boat. Also, when a team member is training one’s colleagues on a subject she knows better than even the more senior colleagues, it increases respect for her professional capabilities, and affects the team member’s self-esteem in a positive way.
Shared knowledge means safety for the business and flexibility in reacting to new situations. In case of a sudden sick leave or a quick new important assignment of a key person in a project, her colleagues can step in and cover for her. The basic skills and processes in the company should be shared by everyone, but it is important to try to identify the special skills and experience each individual has. The team should ensure that this information can be found somewhere, when the person is not available. The difficulty here is that these pieces of knowledge may be subtle, tacit knowledge, and continuously evolving.
One longer term way of spreading the skills and experiences inside the company is the rotation of jobs, where the employee switches from her own position to another for a period of time. This enforces collaborative learning. On the one hand, in the consulting business this is easier to arrange than elsewhere because of the relatively short projects and the variation of the contents from project to project. So, one learns new skills by working on new kinds of projects, typically with a colleague with previous experience on the topic. On the other hand, however, our customers usually do not want us to change the trusted consultants they are used to working with. This is because of the learning curve, of course. In long-term cooperation, however, consultant rotation would be in the customer’s best interest, as it increases the number of available capable resources.
Roles, responsibilities, and delegation
When a small team grows, it is important for the manager to delegate responsibilities to get things done. The manager, as a domain specialist, usually works in projects together with her colleagues. At first, most things outside actual project work can be handled by the manager. Once the team and the workload have grown a bit, more and more of the manager’s time will be dedicated to managerial tasks, which at some point leads to stepping out of project work. Leaving the “real work” behind may be a hard decision for a specialist. This new setup works for some time, but when the growth of the team continues, the manager inevitably becomes an obstacle to efficient function, unless she starts to delegate the managerial responsibilities and decision-making powers to her colleagues. Delegation of responsibilities also releases the powers of others, leading to better usage of all the talent in the team. It is also a good motivational boost for the other team members to have new responsibilities and freedom in decision making. It doesn’t necessarily require a formal higher position.
The delegation of tasks and responsibilities is closely related to the question of clarity in roles and responsibilities. The roles and responsibilities should be clearly defined and not overlapping to avoid ambiguity. At first, it may be reasonable to keep the economic, business, and budget responsibilities in the hands of the manager. For instance, when the team is divided into new smaller teams some people from the team need to be nominated as team leaders, as the direct supervisors of the other team members. They may be responsible for allocating resources into projects and for handling HR issues, but it may not be necessary to immediately make them responsible for the budget of the team, revenues, and costs such as salaries. In general, the delegated tasks should come with decision-making power and some budget, so that the newly nominated person has the tools and the authority for performing the tasks.
Freedom and control, targets, and communication
When delegating, the manager needs to give freedom, but keep control. It is human for anyone to overvalue the issues you get to be in charge of. If your job is to take care of employees’ training and tools, you probably want the company to invest in those areas, instead of, for
example, marketing — and the other way round. There may be a lot of good development ideas and very reasonable investment targets that are certainly important for the company, and it is the manager’s responsibility to control and prioritize actions if you can’t have it all.
The selection of actions taken should come from the company strategy, which is manifested in a road map. Employees want to have freedom, but they want the company to have a clear vision, which gives the direction for everyone to go. Everyday decision making should be based on the road map. This may sound easier than it is, in a changing environment. It is important for the practical everyday life of the team to first establish the strategy, but also to review it in new situations and to communicate it. One thing we have learned is that the strategy, the directions chosen, and even the practical-level decisions made cannot be communicated only once — even if there are no changes in them. They must be communicated clearly in the first place, then repeated, and then repeated again. If that is not done, even the people involved in decision making may soon lose focus, question the strategy, and start to ask the same questions again.
5.3. Building an Integrated Information Architecture Practice at Sapient During the Dot-Com Boom
Lillian E. Svec
Program Coordinator and Instructor, Web Design Program, UCSC Extension in Silicon Valley, Santa Clara, CA
Approved by Sapient 11/4/10
Sapient, a business and technology consulting company, acquired Studio Archetype and Adjacency, two Web design-consulting firms, during 1998–1999. In the summer of 1999, Sapient embarked on merging the competencies of the three original companies into one fully integrated offering. To accomplish this, the integration team looked across the joined companies for groups with analogous work practices. Project team members were reorganized into disciplines including Creative, Strategy, and Technology. The Creative Discipline was further organized into specializations or practices. At this time, the Sapient Information Architecture (IA) Practice was born.
Three groups that had been responsible for developing the information architecture and interaction design for Web sites or application interfaces were joined: the User Interface Group at legacy Sapient, which had existed for three years and had its roots in designing client server applications; the Information Design practice from Studio Archetype, which had existed for seven years and was originally more content design oriented; and the Information Architecture team from Adjacency, which had existed for six months and was more Web application oriented. The outcome was that there were about 60 designers spread throughout five offices who were now Sapient IAs.
The problem was that the work practices of the different groups were similar but they were
not the same. Team members were frequently staffed on projects based in offices other than their home office. In San Francisco, we had members of all three legacy organizations working in the same office. IAs from different legacy offices had different terms and techniques. It was confusing for teams not knowing which “flavor” of IA they were going to be working with. It was confusing for the individual IAs, especially when IAs from different legacy organizations worked together on the same project team.
This situation made it difficult to answer the question “What is Information Architecture at Sapient?” It was impossible to ensure quality and consistency, it was unpredictable for teams and clients, and it was hard to bring new people into the practice and to train them. This was a serious problem because we knew that in the coming year we would be tasked with expanding the practice 50–100% and doubling to tripling the number of offices.
The leaders of the legacy IA recognized that they needed to agree on a definition of the practice, a process model, and nomenclature for tasks and deliverables. In the fall, as the newly appointed Global IA Practice Lead, I realized that if any one legacy felt their point of view had not been taken into consideration, we would fail both to get their buy-in and to gain from the value of their experience.
During the integration process three different levels of expertise within the IA practice were recognized. My strategy was to bring together the twelve most experienced IAs — IA Directors —from throughout the company for a four-day offsite. Our high-level objectives were to learn about the work that each office had done, to craft a common IA Practice definition and process model, and to forge a leadership team across the company. I was fortunate that the Creative Discipline’s executives saw the value in this effort and were willing to allocate budget for it.
In early December, at the offsite, the IA Directors from each office presented the IA process followed in that office and showed “best practice” examples of their deliverables. Using a workshop approach similar to how we worked with clients, we worked as a team, in real time, to develop a detailed IA process model. As each director presented, we used different colored Post-its® for that office’s deliverables and placed them within Sapient’s five-stage process model, which was drawn on a whiteboard. Once all offices had presented, the participants used a variation of the affinity diagramming technique to group the Post-its® into higher level tasks or deliverables and labeled them. Using multiple whiteboards, the group iterated on the emerging model. We identified critical path activities, sub-tasks, deliverables, and entry and exit criteria for each high-level activity and added these to an evolving diagram. At the end of four days and many iterations, we had drafted a comprehensive IA process model, identified “best practice examples” of deliverables, defined the communication vehicles for documenting our work, identified key next steps, and assigned sub-teams to finish the work.
In follow-up work, we wrote a practice definition; defined terms; refined the model and designed a detailed diagram representing it; and created a 122-page, spiral-bound book to document our work. The book included step-by-step best practices guides keyed to the diagram and pointers to examples of deliverables, which were available electronically. Benefiting from a slowdown in our consulting work at the end of 1999 due to the business world’s preoccupation with Y2K information technology fixes, we used the time to conduct these practice development efforts.
As consultants, we recognized that adapting to the needs of each project and to emmerging technologies was crucial. Thus, we were leery of this work being used rigidly and inhibiting flexibility and innovation. For this reason, the work was named the Sapient Information Architecture Framework 1.0. We rolled it out to the IA team members as a guide, not an operating standard, and recognized that it would need to evolve over time.
What we accomplished was an example of the whole being greater than the sum of its parts. We found that the variations in practices between the offices, when synthesized, made for a more comprehensive process than any one office was following.
The Sapient IA practice grew to approximately 100 members in 18 offices worldwide in the next year. The IA Framework was an invaluable aid in this effort and was the foundation we used to develop a weeklong IA training program for new IA hires and cross-training other team members.
Since 1999, the Internet consulting industry and Sapient have changed in countless ways. The company has adopted different organizational structures, work processes, and roles. The Information Architecture role continues to be an important part of the Creative group at SapientNitro, a division of Sapient. The pioneering effort that established this role in the company remains a valuable example of company process integration and codification (see
Figure 5.10).
3Bibliography: Sapient Information Architecture: Practice Definition and Process Framework. March 6, 2000, Version 1.0.Svec, L. (2000). Building an integrated Information Architecture Practice at Sapient. Handout and presentation of the same name, AIGA Advance for Design Summit, Telluride, CO.Svec, L. (2001). Building and Expanding an Integrated Information Architecture Practice. Presentation, ASIS&T IA Summit conference, San Francisco, CA, February 2001.Morrogh, E. (2003). Information architecture: An emerging 21st century profession (pp. 119–121). Upper Saddle River, NJ: Prentice Hall.