Notes from meeting held at ESRF, 20-21st November 2001

A meeting took place at the ESRF involving the main people presently responsible for implementation of the expert system.

The aim of the meeting was to discuss and resolve technical details where possible, to review progress to date and ensure further progress of the project. The meeting started on the morning of the 20th and ended at 11:00 on the 21st.

Business for this meeting

Initial discussions identified the following items for further discussion.

Using as much of Tuesday afternoon as possible the meeting would then be taken up with a practical coding session.

Review of progress

Overall Configuration

The meeting discussed co-ordination of the relevant parts of the overall system.

The data acquisition system will initially make a characterise crystal request of the expert system. At this point it is acting as a client of the expert system. The expert system then requires some images so it makes a request as a client of the data acquisition. At the ESRF a separate data collection server exists and the expert system data collection requests could go direct to this, rather than through the data acquistion GUI. However in the first instance it will work this way i.e. through ProDC. At Daresbury the detector and camera objects are presently built in to the data acquisition system. However these are planned to be separate networked objects but there could also be a networked data collection server. The data acquisition should tell the expert system when these images are available, the expert system then makes an index request to a data processing processing server, in the case the Mosflm server. This naming was seen as important. In future, where appropriate, we should refer to a generic data processing server. In initial project stages this will be taken to be the Mosflm server. However in the future this could well be a different server, supporting the same protocol, to say Denzo.

As the project develops the flow of control could change. The meeting did discuss whether the expert system might have its own GUI which could become the main point of control. However there would still be a requirement for standard manual data collection, beam line set up etc. These features would not be supported by the expert system. No resolution could be reached on future developments along these lines and will have to be returned to in the future.

The meeting agreed that the data processing server should implement the index, mosaicity and strategy steps as individual steps rather than combining more than one into individual steps. This should enable users to follow these separately without having to wait too long for the whole thing to complete. There is also no point in say doing mosaicity calculations in the indexing fails. The ESRF may also want to use Ravelli's strategy program at some stage also. The flow of control is thus defined as in the following UML activity diagram.

XML definitions

The most up to date version of the XML definitions can now be found here.

There was a fairly lengthy discussion of the XML associated with COLLECT commands to the data collection software and INDEX commands to the data processing server. A potential problem in stability of the XML was identified. Perhaps it is also better to modularize the XML and define different areas in different DTD files (e.g. that associated with communications to/from the expert system and that associated with communications to/from Mosflm).

XML was defined as follows:-

The initial characterise crystal request to the expert system

<characterise_crystal>
  <id = "1">
  <symmetry>
  <resolution="2.0">
</characterise_crystal>

Results of the various expert system stages will be relayed to the data collection in the following form. In this manner the output will be formatted by the expert system for any particular data acquisition package to relay to the user, avoiding the need for each to extract output and format messages separately.

<expert_result>
  <status flag="ok|warning|error">
    The indexing succeeded with results x, y, z.
  </status>
</expert_result>

Additional parameters will be required by the expert system to define the following

<beamline_params>
  <phi_now="0.0">
  <wavelength="0.978">
  <distance="250.0">
  <beam_info>
  <slits>
</beamline_params>
<dc_params>
  <exposure_time="1.5">
  <oscillation_range="1.0">
  <no_passes="1">
  <project_id="1">
</dc_params>
<file_info>
  <directory="/data/dna">
  <filename_template="steve_###.img">
</file_info>

Error handling and reporting

The general area of errors was discussed and how to relay these up the chain. Mosflm can presently produce errors in categories such as

It was decided that the data processing server can relay success and error messages in a similar manner to that discussed above, using the following XML

<dp_result>
  <status flag="ok|warning|error">
    Data processing failed : image not found.
  </status>
</dp_result>

There will also be a requirement to abort expert system operations and from there the data processing server and Mosflm itself. A simple XML document was defined to allow this.

<abort/>

Whilst this is not ideal there seemed to be no other way. Details of how the abort might happen were discussed. Also what might happen if a process in the chain were to hang. Unlikely as it may seem there may be a need for processes to be killed in these circumstances. Processes should keep data to recover at a given point if they don't already. There may be a requirement for some separate external process to monitor for process hangs. For now the expert system will monitor progress and determine when to signal timeout errors. The data processing server will have to relay data to the expert system after each image and then wait for commands to continue or stop.

Mosflm server configuration

The configuration of the Mosflm server was discussed. There was some concerned expressed as to stability and updating etc. DL has had problems getting it to work at Daresbury and it took some time to get up at the ESRF. It is still early days in the development of this software and perhaps this is not unexpected. However GW took on board a number of suggestions that are mentioned as specific actions below. It was also felt to be important that multiple versions of Mosflm did not propagate for too long. Indeed this should only be for as little time as possible since the longer things are apart the harder it will be to reconcile differences.

Practical Session

During the latter part of Tuesday afternoon some time was available for programming and other practical work. GW packaged the latest Mosflm server, shipped this to Daresbury. DL and GW then tested this. OS and DS demonstrated ProDC communicating with the dummy expert system. OS gave SK the dummy expert system and SK started work getting Java talking XML over http to the dummy expert system. 

Automation of BM30a

DL brought up a paper he had recently found describing automation of data acquisition and processing already in place on BM30a (J.-L.Ferrer, Acta Cryt. (2001) D57 1752-1753).  On the Wednesday morning we tried to contact staff on BM30 but the relevant people were not at the ESRF on that day. OS and DS will follow this up after the meeting.

Actions

  1. SK - Type up minutes of this meeting, circulate and release.
  2. SK - Update DNA minutes. Create a new section showing the most up to date XML.
  3. GW - Document Mosflm server usage, including examples, a specification of I/O to/from the Mosflm server and a short description of communications to Mosflm.
  4. OS/DS/SK - Continue development of expert system and data acquisition.
  5. GW - Continue development of Mosflm server, particularly relevant here is its use by the expert system.
  6. GW - follow up use of CVS for Mosflm and Mosflm server, together with version information in software and the use of tags.
  7. GW - Implement a system to allow other dna-dev members to get latest Mosflm server source e.g. CVS or tarballs via anonymous ftp.
  8. DL - Advise GW on generation of ChangeLog from CVS.
  9. GW - Make binaries of libraries required by Mosflm server and a build script for them.
  10. OS/DS - Follow up automation on BM30a.
  11. SK - Follow up location for next full DNA meeting.