Backscatter Gauge MSA

Since the capability of the backscatter gauge used to measure thickness has not been assessed recently, the team members decide to perform their first MSA on this measurement system. They learn that typically only one gauge is used, but that as many as 12 operators may use it to measure the anodize thickness of the parts.

You realize that it is not practical to use all 12 operators in the MSA. Instead, you suggest an MSA using three randomly selected operators and five randomly selected production parts. You also suggest that each operator measure each part twice, so that an estimate of repeatability can be calculated. The resulting MSA design is a typical gauge repeatability and reproducibility (R&R) study, with two replications, five parts, and three operators. (Such a design can easily be generated using DOE > Full Factorial Design.) Note that, since the operators are randomly chosen from a larger group of operators, the variation due to operator will be of great interest.

To prevent any systematic effects from impacting the study (equipment warmup, operator fatigue, etc.), the study is run in a completely random order.

The data table ThicknessMSA_1.jmp contains the run order and results. The first ten rows are shown in Exhibit 7.4. Thickness is measured in thousandths of an inch.

The variability in measurements between operators (reproducibility) and within operators (repeatability) is of primary interest. You decide to use a variability chart, also called a Multi-Vari chart¹ to better understand this variation (see Exhibit 7.5).

The chart for Thickness shows, for each Operator and Part, the two measurements obtained by that Operator. These two points are connected by range bars, and a small horizontal dash is placed at the mean of the two measurements. Below the chart for Thickness is a chart showing the standard deviations of the two values for each Operator and Part combination.

Since there are only two values for each Operator and Part combination, and since the range is illustrated in the Thickness chart, you decide to remove the Std Dev chart. You also want to add some additional visual information to the Thickness chart: the group means, the overall mean, and lines connecting the cell means.

The resulting variability chart is shown in Exhibit 7.6. The chart shows that, for the five parts measured, Thickness values range from about 0.22 (Carl's lowest reading for Part 4) to around 2.00 (Lisa's highest reading for Part 3). However, this variation includes Part variation. In an MSA, the focus is on the variation inherent in the measurement process itself rather than on studying part-to-part variation.

Accordingly, the team members turn their focus to the measurement process. They realize that Thickness measurements should be accurate to at least 0.1 thousandths of an inch, since they need to be able to measure in a range of 0.0–1.5 thousandths of an inch. The Thickness plot in Exhibit 7.6 immediately signals that there are issues.

• Measurements on the same Part made by the same Operator can differ by 0.20–0.45 thousandths of an inch. To see this, look at the vertical line (range bar) connecting the two measurements for any one Part within an Operator, and note the magnitude of the difference in the values of Thickness for these two measurements.
• Different operators differ in their overall measurements of the parts. For example, for the five parts, Carl gets an average value slightly over 1.0 (see the solid line across the panel for Carl), while Lisa gets an average of about 1.4, and Scott averages about 1.2. For Part 1, for example, Carl's average reading is about 0.9 thousandths (see the small horizontal tick between the two measured values), while Lisa's is about 1.5, and Scott's is about 1.3.
• There are differential effects in how some operators measure some parts. In other words, there is an Operator by Part interaction. For example, relative to their other part measurements, Part 2 is measured high by Carl, low by Lisa, and at about the same level as the four other parts by Scott.

Even without a formal analysis, the team knows that the measurement process for Thickness must be improved. To gain an understanding of the situation, three of the team members volunteer to observe the measurement process, attempting to make measurements of their own and conferring with the operators who routinely make Thickness measurements.

These three team members observe that operators have difficulty repeating the exact positioning of parts that they measure using the backscatter gauge. Moreover, the gauge proves to be very sensitive to the positioning of the part being measured. They also learn that the amount of pressure applied to the gauge head on the part affects the thickness measurement. In addition, the team members notice that operators do not calibrate the gauge in the same manner, which leads to reproducibility variation.

Armed with this knowledge, the team and a few of the operators work with the metrology department to design a fixture that automatically locates the gauge on the part and adjusts the pressure of the gauge head on the part. At the same time, the team works with the operators to define and implement a standard calibration practice.

Once these changes are implemented, the team conducts a new MSA study to see if they can confirm improvement. Three new operators are chosen for this study. The file ThicknessMSA_2.jmp contains the results (the saved script is Variability Chart). The variability chart for this new study is shown in Exhibit 7.7.

For comparison, you would like to see this new variability chart with the same axis scaling as is used in the chart shown in Exhibit 7.6. To accomplish this, you copy axis settings from the initial variability chart and paste these settings into the new chart.

The resulting chart (shown in Exhibit 7.8) now has the same vertical scale as used in the plot for the initial study. The plot confirms dramatic improvement in both repeatability and reproducibility. In fact, the repeatability and reproducibility variation is not discernable given the scaling of the chart.

The team follows this visual analysis with a formal gauge R&R analysis. Although there are no specification limits for Thickness, the gauge R&R analysis allows the team to estimate the variability in the improved measurement system.

Recall that the measurement process should be able to detect differences of 0.1 thousandths of an inch. The resulting analysis is shown in Exhibit 7.10. It indicates a Gauge R&R value of 0.0133. This means that the measurement system variation, comprised of both repeatability and reproducibility variation, will span a range on the order of only 0.0133 thousandths of an inch.

This indicates that the measurement system will easily distinguish parts that differ by 0.1 thousandths of an inch. Thanks to the new fixture and new procedures, the Thickness measurement process is now extremely capable!

Having completed the study of the backscatter gauge, you guide the team in conducting an MSA on the spectrophotometer used to measure color. Using an analysis similar to the one above, the team finds the spectrophotometer to be extremely capable.

Visual Color Rating MSA

At this point, you address the visual inspection process that results in the lot yield figures. Parts are classified into one of three Color Rating categories: Normal Black, Purple/Black, and Smutty Black. Normal Black characterizes an acceptable part. Purple/Black and Smutty Black signal defective parts, and these may result from different sets of root causes. So, not only is it important to differentiate good parts from bad parts, it is also important to differentiate between these two kinds of bad parts.

You work with the team to design an attribute MSA for the visual inspection process. Eight different inspectors are involved in inspecting color. Three inspectors are randomly chosen as raters for the study. The team chooses 50 parts from production, structuring this sample so that each of the three categories is represented at least 25 percent of the time. That is, the sample of 50 parts contains at least 12 each of the Normal Black, Purple/Black, and Smutty Black parts. This is so that accuracy and agreement relative to all three categories can be estimated with similar precision. Each part is uniquely labeled, but the labels are not visible to the raters.

To choose such a sample and study the accuracy of the visual inspection process, the team identifies an in-house expert rater. Given that customers subsequently return some parts deemed acceptable by Components Inc., the expert rater suggests that he also work with an expert rater from their major customer to rate the parts to be used in the MSA. For the purposes of the MSA, the consensus classification of the parts by the two experts will be considered correct and will be used to evaluate the accuracy of the inspectors.

The experts rate the 50 parts that the team has chosen for the study. The data are given in the table AttributeMSA_PartsOnly.jmp. You use Analyze > Distribution to create a distribution of the color ratings (see Exhibit 7.11; the saved script is Distribution). This confirms that all three categories are well represented, and you deem the 50-part sample appropriate for the study.

Your team randomly selects Hal, Carly, and Jake to be the three raters. Each rater will inspect each part twice. To minimize the potential for recall, the parts will be presented in random order to each rater on two consecutive days. The random presentation order is shown in the table AttributeMSA.jmp. The Part column shows the order of presentation of the parts on each of the two days. The order for each of the Days differs, but to keep the study manageable, the same order was used for all three raters on a given day. Ideally, the study would have been conducted in a completely random order. Note that the Expert Rating is also given in this table.

The team conducts the MSA and records the rating for each rater in the AttributeMSA.jmp table. Part of the analysis of this data is shown in Exhibit 7.12.

You first focus on the kappa criterion in the Agreement Comparisons panel to evaluate interrater agreement, as well as agreement with the expert (these results are shown toward the bottom of the report window). Kappa provides a measure of beyond chance agreement. It is generally accepted that a kappa value between 0.60 and 0.80 indicates substantial agreement, while a kappa value greater than 0.80 reflects almost perfect agreement.

In the Agreement Comparisons panel, you see that all kappa values exceed 0.60. For comparisons of raters to other raters (the top report, in Exhibit 7.12) kappa is always greater than 0.72. The kappa values that measure rater agreement with the experts all exceed 0.80 (the middle report in Exhibit 7.12).

Next, you observe that the Agreement within Raters panel (bottom report in Exhibit 7.12) indicates that raters are fairly repeatable. Each rater rated at least 80 percent of parts the same way on both days.

The effectiveness of the measurement system is a measure of accuracy, that is, of the degree to which the raters agree with the experts. Loosely speaking, the effectiveness of a rater is the proportion of correct decisions made by that rater. An effectiveness of 90 percent or higher is generally considered acceptable. The Effectiveness Report (under the disclosure icon) reports the effectiveness of the measurement system (Exhibit 7.13).

There is room for improvement, as Hal has an effectiveness score below 90 percent and Carly is at 91 percent. Also since these three raters are a random selection from a larger group of raters, it may well be that other raters not used in the study will have effectiveness scores below 90 percent as well. The Misclassifications table gives some insight on the nature of the misclassifications.

Based on the effectiveness scores, you and the team take note that a study addressing improvements in accuracy is warranted. You include this as a recommendation for a separate project. However, for the current project, the team agrees to treat the visual Color Rating measurement system as capable.

Summarizing progress to date, the team has validated that the measurement systems for the four CTQ Ys and for yield are capable. You can now safely proceed to collect and analyze data from the anodize process.

Baseline

The anodize process is usually run in lots of 100 parts, where typically only one lot is run per day. However, occasionally, for various reasons, a complete lot of 100 parts is not available. Only those parts classified by the inspectors as Normal Black are considered acceptable. The project KPI, process yield, is defined as the number of Normal Black parts divided by the lot size, that is, the proportion of parts that are rated Normal Black.

The baseline data consist of two months' worth of lot yields, which are given in the file BaselineYield.jmp. You have computed Yield in this file using a formula. To view this formula in the Formula Editor, click on the plus sign next to Yield in the Columns panel (Exhibit 7.14). To display Yield as a percentage, you apply the Percent format in the Column Info window.

It might appear that Yield, which is intrinsically a proportion, would be monitored by a p chart. However, the process is not likely to be purely binomial (with a single, fixed probability for generating a defective part). More likely, it will be a mixture of binomials because there are many extraneous sources of variation. For example, materials for the parts are purchased from different suppliers, the processing chemicals come from various sources and have varying shelf lives, and different operators assemble the parts. All of these contribute to the likelihood that the underlying proportion defective is not constant from lot to lot.

For this reason, you choose to display the baseline data on an individual measurement chart.

Your team is astounded to see that the average process yield is so low—the average yield is 18.74 percent. The process is apparently stable, except perhaps for an indication of an upward shift starting at lot 34. In other words, the process is producing this unacceptably low yield primarily as a result of common causes of variation, namely, variation that is inherent to the process. Consequently, improvement efforts will have to focus on common causes. In a way, this is good news—it should be easy to improve from such a low level. However, your team's goal of a yield of 90 percent or better is a big stretch!

Data Collection Plan

At this point, you and your team engage in serious thinking and animated discussion about the direction of the project. Color Rating is a visual measure of acceptability given in terms of a nominal (attribute) measurement. You realize that a nominal measure does not provide a sensitive indicator of process behavior. This is why you focused, right from the start, on Thickness and the three color measures, L*, a*, and b*, as continuous surrogates for Color Rating.

Your long-range strategy is this: Your team will design an experiment to model how each of the four continuous Ys varies as a function of various process factors. Assuming that there are significant relationships, you will find optimal settings for the process factors. But what does “optimal” mean? It presumes that you know where the four responses need to be in order to provide a Color Rating of Normal Black.

No specification limits for Thickness, L*, a*, and b* have ever been defined. So you realize that the team also needs to collect data on how Color Rating and the four continuous Ys are related. In particular, you want to determine if there are ranges of values for Thickness, L*, a*, and b* that essentially guarantee that the Color Rating will be acceptable. These ranges would provide specification limits for the four responses, allowing you and, in the long term, production engineers to assess process capability with respect to these responses.

You decide to proceed as follows. Team members will obtain quality inspection records for lots of parts produced over the past six weeks. For five randomly selected parts from each lot produced, they will research and record the values of Color Rating, Thickness, L*, a*, and b*.

It happens that 48 lots were produced during that six-week period. The team collects the data on five parts from each of these 48 lots, resulting in a data table, Anodize_ColorData.jmp, that contains 5 × 48 = 240 rows.

Using Distribution

To begin the process of uncovering relationships, you construct plots for Color Rating and for each of Thickness, L*, a*, and b* using the Distribution platform (Exhibit 7.16).

The distribution of Color Rating shows a proportion of good parts (Normal Black) of about 20 percent. This is not unexpected, given the results of the baseline analysis. However, you are mildly surprised to see that the proportion of Smutty Black parts is about twice the proportion of Purple/Black parts. You also notice that the distributions for Thickness, L*, a*, and b* show clumpings of points, rather than the expected mound-shaped pattern.

You would really like to see the values of Thickness, L*, a*, and b* stratified by the three categories of Color Rating. There are many ways to do this in JMP. Start with the simple approach of clicking on the bars in the bar graph for Color Rating. When you click on the bar for Smutty Black, the 126 rows corresponding to Smutty Black parts are selected in the data table and JMP shades all open histograms to represent these 126 points. Studying the graphs in Exhibit 7.17, you begin to see that only certain ranges of values correspond to Smutty Black parts.

Next, click on the Purple/Black bar. The shaded areas change substantially (Exhibit 7.18). Again, you see that very specific regions of Thickness, L*, a*, and b* values correspond to Purple/Black parts.

However, you are most interested in which values of Thickness, L*, a*, and b* correspond to Normal Black parts. Click on Normal Black in the Color Rating distribution graph (Exhibit 7.19). Note that there is a specific range of values for each of the four responses where the parts are of acceptable quality. In general, Normal Black parts have Thickness values in the range of 0.70 to 1.05 thousandths of an inch. For Normal Black parts, you note that L* values range from roughly 8.0 to 12.0, a* values from 0.0 to 3.0, and b* values from −1.0 to 2.0.

Using Graph Builder

You realize that it would be more efficient to see these distributions in a single display. The JMP Graph Builder is an interactive graphing platform for exploring many variables at a time, with zones for dragging and dropping variables and element icons for displaying various types of graphs. The Graph Builder template is shown in Exhibit 7.20.

Start by exploring how the values of Thickness differ across the three categories of Color Rating (Exhibit 7.21).

The graph stratifies the Thickness measurements according to the three levels of Color Rating: Normal Black, Purple/Black, and Smutty Black. It is easy to see that the distribution of Thickness differs across the three color ratings.

You would like to see similar graphs for all four responses. There are a number of ways of doing this. One approach is to use the Column Switcher (select Scripts > Column Switcher from the red triangle menu) to view each of the four responses in turn. Another approach is to simply add all of the responses to the graph. You opt for the latter approach, which results in one display containing all four responses (Exhibit 7.22).

You study the resulting plot (in Exhibit 7.22) and conclude that Normal Black parts and Purple/Black parts generally appear to have distinct ranges of response values, although there is some overlap in L* values. Normal Black parts and Smutty Black parts seem to share common response values, although there are some systematic tendencies. For example, Normal Black parts tend to have lower Thickness values than do Smutty Black parts.

Although the dot plots reveal the differences in the distributions across the categories of Color Rating, a different graph type, such as a histogram or box plot, might be a better tool for highlighting these differences. Exhibit 7.24 shows box plots for each response across the three color ratings.

This is a compact way to view and present the information that you and your team visualized earlier by clicking on the bars in the bar graph for Color Rating. It shows the differences in the values of the four response variables across the categories of Color Rating, all in a single plot.

Using Scatterplot Matrix

A Scatterplot Matrix (found in the Graph menu) is another tool that might help in the effort to define specification ranges for the four Ys. This graph allows you to explore the color ratings across pairs of response variables (see Exhibit 7.25).

When you click on the text Normal Black in the legend (Exhibit 7.26), the points that correspond to Normal Black in all the scatterplots (the circles) are highlighted and are colored bright red (on a computer screen). Using the legend, click on each Color Rating in turn to highlight the corresponding points in the graphs. (To deselect points, click in the white space in any scatterplot.)

The regions that define each value of Color Rating are even more striking than when viewed in the histograms or box plots. The Purple/Black parts occur in very different regions from Normal Black and Smutty Black parts. More interestingly, whereas the Normal Black and Smutty Black parts were difficult to distinguish using single responses, in the scatterplot matrix you see that they seem to fall into fairly distinct regions of the b* and L* space. In other words, joint values of b* and L* might well distinguish these two groupings.

Using a Scatterplot 3D

The regions that differentiate Color Rating become even more striking when viewed in three dimensions. Scatterplot 3D (found in the Graph menu) provides this three-dimensional view of the data.

Exhibit 7.27 shows the plot with Thickness, L*, and a* on the axes, and with points showing colors and markers for the different color ratings. Recall that these were saved to the data table when we selected the Row Legend earlier.

To explore where the color ratings fall, you use a local data filter for Color Rating. This works like a row legend, allowing you to display points on the Scatterplot 3D corresponding to selected color rating.

In Exhibit 7.28 the points corresponding to Normal Black (again shown by red circles) are displayed using the local data filter, while the points corresponding to Purple/Black and Smutty Black are hidden. You rotate the plot to get a better idea where the Normal Black points fall with respect to Thickness and a*.

As you rotate the plot and use the Local Data Filter to explore the four responses, you can see patterns emerge. It seems clear that Color Rating values are associated with certain ranges of Thickness, L*, a*, and b* as well as with multivariate functions of these values.

Proposing Specifications

Using the information from the histograms and the two- and three-dimensional scatterplots, you feel comfortable in proposing specifications for the four Ys that should generally result in acceptable parts. Although the multidimensional views suggest that combinations of the response values successfully distinguish the three Color Rating groupings, you decide that, because they are easier to work with in practice, you will propose specification limits for each Y individually.

Exhibit 7.29 summarizes the proposed targets and specification ranges for the four Ys. The data indicate that these will generally distinguish Normal Black parts (good parts) from the other two groupings (bad parts), and in particular, from the Smutty Black parts. (You might like to check these limits against the appropriate three-dimensional scatterplots!)

We note that, at this point, you could have used more sophisticated analytical techniques such as discriminant analysis and logistic regression to further your knowledge about the relationship between Color Rating and the four CTQ variables. However, the simple graphical analyses provided you and the team with sufficient knowledge to move to the next step.

Developing the Design

You now face a dilemma in terms of designing the experiment, which must be performed on production equipment. Due to the poor yields of the current process, the equipment is in continual use by manufacturing. Negotiating with the production superintendent, you secure the equipment for a single production shift, during which the team will be allowed to perform the experiment. Unfortunately, at most 12 experimental trials can be performed on a single shift (assuming that the team works very efficiently).

A two-level factorial treatment structure for the five factors (a 2⁵ design) would require 32 runs. Obviously, the size of the experiment needs to be reduced. You consider various options.

Your first thought is to perform a 2^5–2 fractional factorial experiment, which has 8 runs and is a quarter-fraction of the full factorial experiment. With the addition of two center runs, this experiment would have a total of ten runs. However, you realize with the help of the JMP Screening Design platform (DOE > Screening Design) that the 2^5–2 factional factorial is a resolution III design, which means that some main effects are confused with, or aliased with, the joint effect of two factors, which makes it impossible to tell them apart. In fact, for this particular design, each main effect is aliased with a two-way interaction.

You discuss this idea with your teammates, but they decide that it is quite possible that there are two-way interactions among the five factors under study. As a result, you determine that a resolution III fractional factorial design is not the best choice. As you also point out, with five experimental factors there are ten two-way interaction terms. You would need at least 16 runs to estimate the 5 main effects, the 10 interactions, and the intercept in any model. However, due to the limit of 12 runs, such a 16-run design is not feasible, so an appropriate compromise is needed.

You decide to continue this discussion with two experts, who join the team temporarily. Recall that the factors to be studied relate to two steps in the anodize process (Exhibit 7.3):

• Anodize Tank, where the anodize coating is applied
• Dye Tank, where the coated parts are dyed in a separate tank

The two experts maintain that interactions cannot occur between the two dye tank factors and the three anodize tank factors, although two-way interactions can certainly occur among the factors within each of the two steps. If the team does not estimate interactions between factors from the two anodize process steps, only four two-way interactions need to be estimated:

• Anodize Temperature*Anodize Time
• Anodize Temperature*Acid Concentration
• Anodize Time*Acid Concentration
• Dye Concentration*Dye pH

It is possible to estimate the five main effects, the four two-way interactions of importance, and the intercept in a model with only 10 runs. Given the 12-run limitation, this design is feasible. In fact, you can add two runs. If the effects are large relative to the error variation, the resulting design is likely to identify them. You are reasonably comfortable proceeding under the experts' assumption, realizing that any proposed solution will be verified using confirmation trials before it is adopted.

Another critical piece of knowledge relative to designing the experiment involves the actual factors settings to be used. With the help of the two experts that the team has commandeered, low and high levels for the five factors are specified. You are careful to ensure that these levels are aggressive relative to the production settings. The thinking is that, if a factor or interaction has an effect, you want to maximize the chance of detecting it.

You now proceed to design the experiment. The design requirements cannot be met using a classical design, so you generate a Custom Design (from the DOE menu). The Custom Design platform allows you to specify a constraint on the total number of trials and to specify the effects to be estimated. The platform then searches for an optimal design that satisfies your requirements.² The custom design window, with the four responses and five factors, is shown in Exhibit 7.30.

Initially, the Model panel shows the list of main effects. You add the required four two-way interactions manually. Then, you indicate the number of runs. You request 12 runs in all, reserving two runs for center points to provide an estimate of repeatability. The completed dialog is shown in Exhibit 7.31.

The design appears in the Design panel (see Exhibit 7.32). It is constructed so that the runs are in random order. Note that the two center points appear as runs 10 and 12.

The design that you obtain will very likely differ from the one shown in Exhibit 7.32. This is because the algorithm used to construct custom designs requires a random seed to determine a starting design. The algorithm then runs for a fixed number of seconds, during which a number of designs are constructed based on random starts. The final design is the best one found, based on the optimality criterion.

The script for the model that you will eventually fit to the data you collect, called Model, is located in the Table panel in the upper left corner of the data table (see Exhibit 7.33). This script defines the model that you specified when you built the design, namely, a model with five main effects and four specific two-way interactions (to see this, run the Model script).

JMP has saved two other scripts to this data table. One of these scripts, Screening, runs a screening analysis that fits a saturated model. You will not be interested in this analysis since you have some knowledge of the model that is appropriate and you have enough observations to estimate error for that model. The other script, DOE Dialog, reproduces the DOE > Custom Design dialog used to obtain this design.

Notice the asterisks next to the variable names in the Columns panel (Exhibit 7.34). When you click on the asterisks, you see that JMP has saved a number of Column Properties for each of the factors: Coding, Design Role, and Factor Changes. Clicking on any one of these takes you directly to that property in Column Info. Similarly, for each response, Response Limits have been saved as a Column Property.

You will use the results of this experiment to identify the optimal settings for the four responses, and then to estimate the capability and PPM values at these settings. So, you add specification limits for each response as a Spec Limits column property. These are based on the values listed in Exhibit 7.29.

Conducting the Experiment

You are now ready to perform the experiment. You explain the importance of following the randomization order and of resetting all experimental conditions between runs, and the team appreciates the importance of these procedures. You plan the details of how the experiment will be conducted, and number the parts produced to mistake-proof the process of taking measurements of the responses.

The design and measured responses from the experiment are given in the data table Anodize_CustomDesign_Results.jmp (Exhibit 7.36).

Uncovering the Hot Xs

It is time to analyze the data and you and your team are very excited! Since JMP has saved the Model script to the data table, you start by running this script. This takes you to the Fit Model Specification window. You see that JMP has included all of the responses and the model effects that you specified when you designed the experiment, so you select Run to fit the specified model to each of the responses.

At the top of the report (in Exhibit 7.37) is an Effect Summary table, which provides a summary of the significance of each of the terms across all of the models. Every term, except the interaction Anodize Temp*Anodize Time, is highly significant (with a PValue < 0.01) in at least one of the models.

Since you are interested in understanding which factors and interactions are significant in predicting each of your four responses, you decide to identify significant effects by modeling each response separately. For each response, you follow this strategy:

• Examine the data for outliers and possible lack of fit, using the Actual by Predicted plot as a visual guide. Check the Lack Of Fit Test, which can be conducted thanks to the two center points, in order to confirm your visual assessment of the Actual by Predicted plot.
• Find a best model by eliminating effects, one at a time, that appear insignificant.
• Save the prediction formula for the best model as a column in the data table.
• Save the script for the best model to the data table for future reference.

Your plan, once this work is completed, is to use the Profiler from the Graph menu to find factor level settings that will simultaneously optimize all four responses.

The report for the model for Thickness, shown in Exhibit 7.38, shows no significant lack of fit—both the Actual by Predicted plot and the formal Lack Of Fit test support this conclusion (the p-value for the Lack Of Fit test is Prob > F = 0.4763). Note that this is a nice example of Exploratory Data Analysis (the Actual by Predicted plot) being reinforced by Confirmatory Data Analysis (the Lack Of Fit test).

Since the model appears to fit the data well, you check the Analysis of Variance table and see that the overall model is significant (Exhibit 7.39). You could examine the Effect Tests tables to see which effects are significant, but you find it easier to interpret the Sorted Parameter Estimates table (Exhibit 7.39), which gives a graph where the size of a bar is proportional to the corresponding effect's significance.

You examine the p-values provided under Prob > |t|, and see that two of the two-way interactions, Anodize Temp*Anodize Time and Dye pH*Dye Conc, do not appear to be significant (you use the 0.05 p-value guideline for significance).

The Effect Summary table at the top of the window (Exhibit 7.39) repeats the p-values shown in the Sorted Parameter Estimates (displaying one additional decimal place). However, this table also provides an interactive way of refining the model. For example, nonsignificant terms can be taken out with the Remove button.

The caret (^) next to a p-value indicates that the corresponding term is involved in one or more interactions with smaller p-values. Significant or not, terms marked with ^ should be retained in the model if they are involved in significant interactions. This practice follows from the principle of Effect Heredity.³ The implication for reducing models is that nonsignificant higher-order terms are removed prior to removing main effects.

At this point, you start reducing the model, one term at a time. Dye pH and Dye Conc are the two least significant terms, but both are involved in the Dye pH*Dye Conc interaction. This interaction is the next least significant term. Remove the Dye pH*Dye Conc interaction term from the model using the Remove button. All of the results in the window automatically update based on the new model. You note that Dye pH is now the next least significant term and it is not a component of an interaction, so you remove it from the model. Next you successively remove Dye Conc and Anodize Temp*Anodize Time. At this point, all remaining terms are significant at the 0.05 level (Exhibit 7.40).

You notice that the final Thickness model contains two significant interactions, and that only factors from the anodizing step of the process are significant. Thus, the team finds that the model is in good agreement with engineering knowledge, which indicates that factors in the dye step should not have an impact on anodize thickness.

You use this same approach to determine final models for each of the other three responses. Scripts that show results for all four models are saved to the Table panel in the data table Anodize_CustomDesign_Results.jmp (see Exhibit 7.41). Each of the models for L*, a*, and b* includes factors from both the anodizing and dyeing processes.

Determining Optimal Factor Level Settings

Now that you have statistical models for the four responses, your intent is to identify settings of the five Xs that optimize these four Ys. You suspect that factor settings that optimize one response are likely to degrade the performance measured by another response. For this reason, it is important that simultaneous optimization be conducted to give a sound measure of overall performance.

In the Analyze Phase, your team defined target values and specification limits for the four Ys, hoping to guarantee acceptable color quality for the anodized parts. Using these targets and specifications as a basis for optimization, you will perform multiple response optimization in JMP.

JMP bases multiple optimization on a desirability function. Recall that, when you entered the responses in the Custom Design dialog, you noted that the goal for each response was to Match Target, and you entered the specification limits as response limits—see the Lower Limit and Upper Limit entries under Responses in Exhibit 7.30. You also assigned Importance values of 1 to each response (also Exhibit 7.30), indicating that the responses are of equal importance. (What is relevant is the ratio of these values; they could equally well have all been assigned as 0.25.)

The desirability function constructs a single criterion from the response limits and importance values. This function weights the responses according to importance, and, in a Match Target situation, places the highest desirability on values in the middle of the response range (the user can manually set the target elsewhere, if desired). The desirability function is a function of the set of factors that is involved in the union of the four models. In this case, since each factor appears in at least one of the models, the desirability function is a function of all five process factors.

In JMP, desirability functions are accessed from the Profiler, which is often called the Prediction Profiler to distinguish it from the several other profilers that JMP provides. The Profiler for a single response can be found in the Fit Model report for that response. When different models are fit to multiple responses, the Profiler can also be accessed from the Graph menu. But, you must first save prediction formulas for each of the responses; otherwise, JMP will not have the underlying models available to optimize.

Each of these columns is defined by the formula for the model for the response specified. For example, the prediction formula for a* is given in Exhibit 7.43.

Once the prediction formulas for the four models have been saved, you can profile the four prediction formulas together. The resulting Profiler is shown in Exhibit 7.44.

The desirability functions for each individual response are displayed in the rightmost column. For each response, the maximum desirability value is 1.0, and this occurs at the midpoint of the response limits. The least desirable value is 0.0, and this occurs near the lower and upper response limits. (Since your specifications are symmetric, having the highest desirability at the midpoint makes sense.) The cells in the bottom row in Exhibit 7.44 show traces, or cross-sections, for the desirability function associated with the simultaneous optimization of all four responses.

Recall that the response limits for b* were −2.0 and +2.0. To better understand the desirability function, you double-click in the desirability panel for b*, in the rightmost column. The resulting dialog is displayed in Exhibit 7.45. You note that −2.0 and +2.0 are given desirability close to 0, namely, 0.0183, and that the midpoint between the response limits, 0, is given desirability 1. If you want to change any of these settings, you can do so in this dialog (click Cancel to close this dialog).

The Profiler is dynamically linked to the models for the responses. When the Profiler first appears, the dotted (red) vertical line in each panel is set to the average predictor value, which is the midpoint of the design interval. By moving the dotted (red) vertical line for a given process factor, one can see the effect of changes on the four responses. This powerful dynamic visualization technique enables what-if inquiries, such as, “What happens if we increase Anodize Time?” The team explores various scenarios using this feature, before returning to the goal of optimization.

To perform the optimization, you select the Maximize Desirability option from the red triangle next to Prediction Profiler. Results for the optimization of the four responses are shown in Exhibit 7.46. However, since there are usually many equivalent solutions to such an optimization problem, the results you obtain may differ from those shown in Exhibit 7.46 (these specific results can be obtained by running the Profiler 2 script in Anodize_CustomDesign_Results.jmp).

A wealth of information concerning the responses and the process variable settings is provided in this visual display. At the bottom of the display, you see optimal settings for each of the five process variables (the figures in red in Exhibit 7.46). These are vastly different from the settings currently used in production. To the left of the display you see the predicted mean response values associated with these optimal settings (also in red).

You note that the predicted mean levels of all four responses are reasonably close to their specified targets and are well within the specification limits. It does appear that further optimization could be achieved by considering higher values of Anodize Time and Acid Conc, since the optimal settings of these variables are at the extremes of their design ranges. You make a note to consider expanding these ranges in a future experiment.

Linking with Contour Profiler

Even though the optimal factor settings obtained are feasible in this case, it is always informative to investigate other possible optimal or near-optimal settings. You believe that the Contour Profiler would be useful in this context. You create a Contour Profiler to explore the four prediction formulas. The report in Exhibit 7.47 shows contours of the four prediction formulas, as well as small surface plots of those formulas.

In the top part of the Contour Profiler report, the Current X values are the midpoints of the design intervals. Recall that these were the initial settings for the factors in the Prediction Profiler as well. You would like to set these at the optimal settings obtained using the Prediction Profiler. You learn that this can be achieved by linking the two profilers. After the profilers are linked, the settings for Current X in the Contour Profiler update to match the optimal settings found in the Prediction Profiler (Exhibit 7.48).

Now, by choosing pairs of Horiz and Vert factors in the top corner of the Contour Profiler, and by moving the sliders next to these or by moving the crosshairs in the contour plot, you can see the effect of changing factor settings on the predicted responses. Since the profilers are linked, you can see the effect on overall desirability by checking the Prediction Profiler, which updates as the factor settings are changed in the Contour Profiler.

The Contour Profiler, with its ability to link to the Prediction Profiler, is an extremely powerful tool in terms of exploring alternative factor level settings. (In fact, all JMP Profilers, including the Surface Plot, link together.) In some cases it might be more economical, or necessary for other reasons, to run at settings different from those found as optimal by the Prediction Profiler. These tools allow you to find practically equivalent or superior alternative settings by assessing the loss in performance relative to the theoretical, but unworkable, optimum.

At this point, you close the Contour Profiler and the Prediction Profiler. Since you have been exploring various settings of the predictors, you rerun the script Profiler 2 to retrieve the optimal settings.

Sensitivity

The Prediction Profiler report provides three ways to assess the sensitivity of the responses to the settings of the process variables: desirability traces, a sensitivity indicator, and Variable Importance. We illustrate the first two methods. The third method, Variable Importance, is an option under the Profiler red triangle menu. For each factor, Variable Importance provides an index of predictive variability that is based on a collection of simulated settings. See the JMP documentation for details.

Notice that the last row of the Prediction Profiler display, repeated in Exhibit 7.49, contains desirability traces for each of the process variables. These traces represent the overall sensitivity of the combined desirability functions to variation in the settings of the process factors. For example, the desirability trace for Anodize Temp is peaked, with sharply descending curves on either side of the peak. Thus, the desirability function is more sensitive to variation in the setting of Anodize Temp than, say, to Dye pH, which is much less peaked in comparison. Variation in the setting of Anodize Temp will cause significant variation in the desirability of the responses.

Click the red triangle in the Prediction Profiler report panel and select Sensitivity Indicator. These indicators appear in Exhibit 7.50 as small triangles in each of the response profiles. (Note that the grabber tool has been used to rescale some of the axes so that the triangles are more visible; when you place your cursor over the ends of an axis, the grabber tool will automatically appear.) The height of each triangle indicates the relative sensitivity of that response at the corresponding process variable's current setting. The triangle points up or down to indicate whether the predicted response increases or decreases, respectively, as the process variable increases.

For Anodize Temp, you notice that Pred Formula L* and Pred Formula a* both have relatively tall downward-pointing triangles, indicating that, according to your models, both L* and a* will decrease fairly sharply with an increase in Anodize Temp. Similarly, you see that Pred Formula Thickness and Pred Formula b* have upward-pointing triangles, indicating that those responses will increase with an increase in Anodize Temp.

You notice the horizontal traces and lack of sensitivity indicators for Dye pH and Dye Conc in the row for Pred Formula Thickness. Remember that not all factors appear in all prediction formulas. Specifically, the dye variables did not appear in the model for Thickness. So, it makes sense that horizontal lines appear, and that no sensitivity indicators are given, since the data lead to a model in which Dye pH and Dye Conc are unrelated to Thickness.

From this analysis, you conclude that the joint desirability of the responses will be quite sensitive to variation in the process variables in the region of their new optimal settings. The team reminds you that some process experts did not believe, prior to the experiment, that the anodize process, and especially color, was sensitive to Anodize Temp. It is because of this unfounded belief that temperature is not controlled well in the current process. The team views this lack of control over temperature as a potentially large contributor to the low yields and substantial run-to-run variation seen in the current process.

Confirmation Runs

The team now thinks it has a potential solution to the color problem. Namely, the process should be run at the optimized settings for the Ys, while controlling the Xs as tightly as possible. The Revise Knowledge step in the Visual Six Sigma Roadmap (see Exhibit 3.30) addresses the extent to which our conclusions generalize. Gathering new data through confirmation trials at the optimal settings will either provide support for the model or indicate that the model falls short of describing reality.

So, to see if the optimal settings actually do result in good product, you suggest that the team conduct some confirmation runs. Such confirmation is essential before implementing a systemic change to how a process operates. In addition to being good common sense, this strategy will address the skepticism of some of the subject matter experts who are not involved with the team.

With support from the production manager, your team performs two confirmatory production runs at the optimized settings for the process variables. The results of these confirmation runs are very favorable—not only do both lots have 100 percent yield, the outgoing inspectors declare that these parts have the best visual appearance they have ever seen. The team also ships some of these parts to Component Inc.'s main customer, who reports that these are the best they have received from any supplier.

Projected Capability

At this point, your team is ready to develop an implementation plan to run the process at the new optimized settings. However, you restrain the team from doing this until the capability of the new process is estimated. You point out that this is very important, since some of the responses are quite sensitive to variation in the process variables.

In Design for Six Sigma (DFSS) applications, estimation of response distribution properties is sometimes referred to as response distribution analysis. Predictive models, or more generally transfer functions, are used to estimate or simulate the amount of variation that will be observed in the responses as a function of variation in the model inputs.

The Simulator

The Prediction Profiler includes an excellent simulation feature. Even when set at their optimal values, most process variables will have some variation in their settings. Also, there is variation in the response that is caused by other process factors. You want to include both of these sources of variation in obtaining capability estimates. Once you have obtained these estimates of variability, you will use the Simulator in the Prediction Profiler to estimate the overall process capability at the new settings.

To obtain information on the variability of factors and responses, your team collects data on batches produced over a two-week period. The five factors are set to their optimal settings. They are measured at periodic intervals to obtain estimates of variation in those settings. Process engineers indicate that you can control Anodize Time with essentially no error, and so you can treat this factor as fixed during simulations. Standard deviations for the other process factors are computed.

The responses are also measured at the same time as the factors. You fit a regression model for each response using the data collected on the factors and responses. The root mean squared error (RMSE) from each model provides an estimate of the standard deviation of the response at the optimal settings. It represents the variation not explained by the five process factors.

These estimates of standard deviation for factors and responses are given in Exhibit 7.51.

Variable	Variable Type	Estimated Standard Deviation
Anodize Temp	Factor	1.542
Acid Conc	Factor	1.625
Dye pH	Factor	0.100
Dye Conc	Factor	0.323
Pred Formula Thickness	Response	0.015
Pred Formula L*	Response	0.200
Pred Formula a*	Response	0.205
Pred Formula b*	Response	0.007

Add these estimates as Sigma column properties for each of the four factors and for the four prediction formulas for the responses. Then return to the Prediction Profiler to run the simulation. When you add the Sigma column property for a factor or prediction formula, the Simulator default is to simulate random values with standard deviations equal to the specified values for Sigma. The default for factors is to set the mean equal to the Profiler setting for the factor.

When you click the Simulate button, JMP simulates 5,000 combinations of factor settings using the specified normal distributions. JMP calculates predicted values plus error variation, as specified by the Std Dev values, for each of the four responses (script is Profiler 4). Histograms in the rightmost column of the Prediction Profiler show the simulated values plotted against their specification limits (see Exhibit 7.53). Keep in mind that your optimal settings may differ from those shown and that simulated results will vary.

Based on the simulation results and using the Spec Limits column properties, the Prediction Profiler calculates estimated defect rates for each of the four responses. The estimated defect rate for L* is 0.0064, or 6,400 parts per million, which is higher than the team would like. For the other three responses, at least to four decimal places, the estimated defect rate is 0.

Simulated Capability

Rerunning the simulation a few times indicates that the overall defect rate is not likely to exceed 0.008, corresponding to a PPM level of 8,000. To obtain an estimate of the capability of the process when run at the new settings, save the 5,000 simulated response values from one of your simulations to a new data table. Click Make Table under Simulate to Table. (See bottom of Exhibit 7.53.) This runs the simulation and creates a new data table with the simulated results and a Distribution script.

Open Anodize_CustomDesign_Simulation.jmp to see a set of simulated results. Run the Distribution script in this data table to see histograms and capability analyses for the four predicted responses. The capability analyses are provided because specification limits were saved as column properties for the original responses and JMP carried those forward to the prediction formulas.

Because the capability analyses are based on randomly simulated data, the results will change slightly each time you simulate. To reproduce the results shown here, use the simulated data in Anodize_CustomDesign_Simulation.jmp.

The report for Pred Formula Thickness is shown in Exhibit 7.54. For Pred Formula Thickness, CPK is 2.66. Note that the process is slightly off center.

You can use JMP's Process Capability platform to visualize capability-related data when several responses are of interest. Although you have only four responses, you are interested in exploring the simulated capability using this platform.

The capability report for the four responses is shown in Exhibit 7.55. The Goal Plot displays a point for each predicted response. The horizontal value for a response is its mean shift from the target divided by its spec range. The vertical value is its standard deviation divided by its spec range. The ideal location for a response is near (0, 0); it should be on target and its standard deviation should be small relative to its spec limits.

The slider at the bottom of the Goal Plot is set by default to a Ppk of 1.0. The slider defines a triangular, or goal, area in the plot, within which responses have Ppk values that exceed 1.0. The slider can be moved to change this threshold Ppk value and the corresponding goal area. Recall that a stable and centered process with a Ppk (long term or overall process capability) of 1.0 has a 0.27 percent defect rate; such a process is generally considered unacceptable. Assuming that a process is stable and centered, a Ppk of 1.5 (corresponding to a one-sided rate of 3.4 defective items per million) is generally the hallmark of an acceptable process.

When a Ppk of 1.5 is entered, the triangular region gets smaller, but three of the four responses still fall in (or on the border of) the goal area. Only Pred Formula L* falls outside the 1.5 Ppk region.

From the red triangle menu at the top level of the report, next to Capability, select Summary Reports > Overall Sigma Report Summary. This report shows summary information for all four predicted responses, as well as their Ppk values (Exhibit 7.57). It is clear that L* would benefit from further study and work.

Note that some default summary information is not displayed in Exhibit 7.57, and that Expected PPM Outside has been added. You can control what appears in the Overall Sigma Capability Summary Report by right-clicking in the report, selecting Columns, and then making selections to add or remove summary information. The saved script is Process Capability 3.

The second plot in Exhibit 7.55 shows Capability Box Plots. These are box plots constructed from appropriately centered and scaled data that allow a fair comparison between responses. Because each of your responses has two specification limits that are symmetric about the target, the box plot is constructed from values obtained as follows: The response target is subtracted from each measurement, and then this difference is divided by the specification range. This scaling places the spec limits at 0.50 and −0.50. You see at a glance that the simulated responses fall off target. In the case of L*, which is the most variable relative to the specification range, some simulated values fall below the lower spec limit.

New Temperature Control System

As you reexamine results from the earlier sensitivity analysis, you recall that L* and a* are particularly sensitive to variation in Anodize Temp, which is not well controlled in the current process (Exhibit 7.51). You suspect that, if the variation in Anodize Temp can be reduced, then conformance to specifications will improve, particularly for L*.

Your team members engage in an effort to find an affordable temperature control system for the anodize bath. They find a system that will virtually eliminate variation in the bath temperature during production runs. Before initiating the purchasing process, the team asks you to estimate the expected process capability if they were to control temperature with this new system.

Conservative estimates indicate that the new control system will cut the standard deviation of Anodize Temp in half, from 1.50 to 0.75. To explore the effect of this change, return to Anodize_CustomDesign_Results.jmp and rerun the script Profiler 4. Change the standard deviation for Anodize Temp at the bottom of the Prediction Profiler panel to 0.75. The saved script is Profiler 5.

Now construct a new simulated data table, and again use the Process Capability platform to obtain capability analyses. The Overall Sigma Capability Summary Report is shown in Exhibit 7.58. Again, since these values are based on a random simulation, the values you obtain may differ slightly. To obtain the same results as in Exhibit 7.58, use the data table Anodize_CustomDesign_Simulation2.jmp. The saved script is Process Capability.

The new capability analyses indicate that Thickness, a*, and b* have extremely high capability values and very low PPM defect rates. Most importantly, the PPM rate for L* has dropped dramatically, and it now has a Ppk value of about 1.074.

The team runs some additional confirmation runs at the optimal settings, exerting tight control of Anodize Temp. Everyone is thrilled when these all result in 100 percent yield! You wonder if perhaps the specification limits for L* could be widened, without negatively impacting the yield. You make a note to launch a follow-up project to investigate this further.

At this point, the team is ready to recommend purchase of the new temperature control system and to begin operating at the settings identified in the optimization. You guide the team in preparing an implementation plan. The team, with your support, reports its findings to a management team. Based on your rigorous approach, and more importantly the projected capability figures, management accepts your recommendations and instructs the team to implement their solution. With this, the project enters the Control Phase.