Chapter 3: Integration and Scaling the Data


Step 13: 3D Window and Mosaicity


Step 13: Integration

Integrate the set of frames by clicking on the integrate button. Monitor the progress of the integration by examining the χ2, Cell Constants, Crystal Rotations, Mosaicity, and Distance vs. Frame plots, as well as the agreement between the predicted reflections and the spots on each image. Ideally, all of these plots will yield horizontal lines.

The Integrate button is red. Is this a problem?

The lettering in the button should be colored green. If it is orange, that means that either the χ2 values are suspiciously high and should be checked, or else you have failed to select a lattice other than P1. If the lettering is red, this warns you that the χ2 values are unacceptably high and should be investigated (Figure 41). You can still continue, but there is probably a problem and you will have to address it eventually.







Figure 41. The Integrate buttons colored green, orange, red

The Crystal Slippage and Special integration buttons.

The Special button opens up the command file used in the integration of the diffraction images. If you want to customize the integration, for example, putting in many more “go” statements or some other modification to the standard procedure; this is the place to make those changes (Figure 42).

    *     You can also make these changes under the Macros tab in the HKL-2000 interface.

The Crystal Slippage button is a shorthand way of performing the integration in two resolution stages. The first stage is done at low resolution (4 Å), and then is expanded to the final resolution. Normally, integration is done to the highest resolution immediately. However, if the crystal is slipping during data collection, the crystal orientation parameters may not be consistent enough from frame to frame to process frames automatically. Using only the low order reflections at first allows a larger radius of convergence, which helps account for slipping crystals.

Figure 42. The special integration option

What are these plots supposed to look like and how do I know if everything is ok?

After clicking, the display will change to a graphical plot of the Integration progress, and show the currently refined mosaicity as well. One of the most diagnostic plots is χ2 vs. Frame (Figure 43). Ideally both lines will be in the range of 1 to 2 over the set of processed frames. All of the plots should be examined, with particular attention paid to detecting bad frames, which will show up as strong outliers in the Integration Information plots. Ideally all of the plots – χ2, Unit Cell Constants, Distance, and Mosaicity – should be flat as a function of frame number. You’ll also notice that the points, for each plot, appear in bunches that correspond to the number of frames in the 3D window. If in the diagnostic plot χ2 vs. Frame some frames will have unreasonable high χ2 it makes sense to exclude those bad frames from scaling. To do this split the data set in two or more groups. 


Figure 43. The diagnostic plots: χ2 vs. Frame and Distance Change vs. Frame

What is an acceptable range for χ2?

Ideally, the χ2 x and y values should be around 1 (i.e. colored green in the Refinement Information panel; see Figure 33). Sometimes they rise up into the 3 – 4 range. If the preds continue to superimpose on the reflections as the integration proceeds, then there is no immediate cause for alarm. However, if you begin refining to high values of mosaicity (greater than 2% for frozen crystals), or the crystal orientation, cell, or distance parameters are changing dramatically, you should consider re-indexing or choosing another lattice. In some particularly problematic integrations, especially when the calculated unit cell parameters are changing substantially, you may try fixing the mosaicity by clicking the Fix Mosaicity button on the Controls panel (Figure 42). This is particularly relevant for cases where there are several lattices in the beam at the same time. In the best-case scenario only one of the lattices would dominate the diffraction pattern and be indexed, but sometimes as the crystal rotates other lattices become more prominent and the program may start to confuse them. By restricting the mosaicity you may be able to effectively limit the number of nearby spots that can be included in the integration.

Houston, we’ve got problems.” The χ2 values are getting larger!

Ideally the χ2 values will be close to 1. χ2 values that go up to 3 and even 4 are sometimes normal and may not indicate a serious problem. However, χ2 values that are off the plot that is displayed during integration probably mean that something is wrong. The most likely culprit, for otherwise “normal” data sets (i.e. where the diffraction pattern on the poorly indexed frames still looks ok, and there weren’t any shutter or goniostat problems) is misindexing due to an incorrect x or y beam value. Experimenting with different x and y beam values may solve the problem. This is relatively easy to do using the set beam position button found in the Index panel. Crystal idiosyncrasies (such as slight splitting, which may not be apparent from an inspection of the diffraction images) or shutter, spindle or goniostat problems can also lead to high χ2 values. Problems with the experimental setup will often manifest themselves suddenly, and a sharp change in the χ2 values will pinpoint the frames that have been corrupted. It is always a good idea to verify the experimental setup by collecting a data set from a known, high quality crystal in order to benchmark your setup.

Why do the bars in the mosaicity histogram get thicker or thinner?

The thickness of the bars is proportional to the number of single partial reflections being integrated in the 3D Window. More reflections will lead to thicker bars.

How long should this take? How do I know if the program is stuck?

When the integration is complete the display will show the message “Integration Complete”. Obviously the speed of the integration depends on both hardware and data-specific factors. Among the hardware ones are: CPU speed, amount of physical memory, and the speed at which the computer can access the raw data from disk (local disks are faster than remote ones). Data-specific factors include the number of observed and predicted reflections on each image, size of the image file (larger frames take longer to read from disk), the number of frames in the 3D Window (integrating 5 frames at a time will take about 5 times as long as integrating a single frame, assuming that you have the physical memory to handle that much in the first place). In my experience the integration plugs along and if several minutes go by with no additions to the plots then something is probably amiss.

Waiting for image frames.

A very nice feature of HKL-2000 is the program’s ability to wait for image frames during processing. This is indicated by the message “Waiting for Frames” which appears above the Integration Information window during integration (Figure 44). The program knows to wait for frames based on the value you entered in the Number of Frames window of the main HKL-2000 page. For example, let’s say you are collecting 60 images for a data set, but only 10 have been collected so far and the others are on the way (or being read from a tape, read from a CD/DVD, coming across over ftp, or whatever). When you first set up the data files, only 10 frames will be seen by the program. However, if you enter 60 for Number of Frames, the program will know that eventually 60 frames will constitute the complete set of diffraction images. You can start the indexing and integration, and these will proceed through the first 10 (or however many you have) frames currently on disk. When those have been integrated, the message “Waiting for Frames” will appear. At this point, you don’t have to do anything. As more frames appear, they will be automatically read and integrated. This is allows you to monitor the progress of your data collection as it occurs. It also allows you to use your data collection time (e.g. at the synchrotron) efficiently.

Figure 44. The program is waiting for new frames during processing the data

Once the integration is complete, should I update the site?

Ordinarily you would not update the site unless:

  1. you have the password to do so,

  2. the refined values of the site parameters (X and Y Beam Positions, Crossfire, Y Scale) are substantially different from the original site values,

  3. the values you’ve refined are reliable.

Site parameters determined from weak, poorly diffracting crystals may not be much of an improvement, if at all, from the default values. Generally you want to use a superior-diffracting crystal to determine updated site parameter values.

How do I stop and start over?

To stop the integration hit abort and you will get a massage box with the question: “Which crystal orientation and cell parameters would you like to keep?” You can choose between three options: “original values” (values as read from the header of data files), “indexed values” (values after last autoindexing round), and “current values”. Now you can reindex if you’d like.



Chapter 3


Table of Contents