Notes from the DNA Meeting

Daresbury Laboratory 1st Feb 2002

Present: Olof Svensson, Solange Delageniere, Darren Spruce, Andrew Leslie, Graeme Winter, Harry Powell, Charles Ballard, Martyn Winn, Steve Kinder, Dave Love, Karen Ackroyd, Liz Duke, Colin Nave, Keith Wilson.

Agenda:

  1. Actions from previous developers’ meeting
  2. Progress at ESRF and demo
  3. Progress at DL and demo
  4. Progress at Cambridge
  5. Issues out of recent progess
  6. Version handling
  7. Data definitions – leading to XML
  8. The way forward
  9. Developers sub-group meeting

Meeting Notes

  1. Actions from previous developers’ meeting
  2. Progress at ESRF and demo
  3. The goal was set at the ESRF to have a system running to "characterise crystal" before Christmas. This was achieved by rapidly writing some code to communicate between the Expert System and ProDC. Olof gave a demonstration. The server, ProDC and the expert system were both started manually. ProDC is a Tcl programme – a graphical interface to underlying systems. There is an automatic interaction between ProDC and all other software on the station including the personnel safety system. A "characterise crystal" button has been put into the ProDC data collection parameters window. Clicking on the "characterise crystal" button causes 2 images to be collected, 1 at 0 – 1o and the other from 90 - 91o . Mosflm is then automatically used to autoindex the two images and the best solution (as chosen by mosflm) is reported back. The system has been successfully demonstrated using several (non lysozyme) crystals.

    There are plans to make the collection of the two images into 1 call from ProDC – currently it is done in 2 calls. The ESRF users are keen to have this facility available as soon as possible and there are plans to make this possible as soon as it can be ensured that all of the software can be made to run automatically.

    The next stage is to have a value for the mosaicity reported back along with a suggested strategy.

  4. Progress at DL and demo
  5. A standalone GUI was successfully written testing communication to the expert system and to the mosflm server. The expert system used was the same as that developed at the ESRF. A java class was written to interface to the expert system. Communication to and from the expert system was done using XML parsing via CASTOR (see http://castor.exolab.org for more information). The mosflm server used was the one produced at the end of November.

    The new GUI for data collection at Daresbury Laboratory, PXGEN++, has a button "characterise crystal". With the expert system and the mosflm server independently started, if the "characterise crystal" button is clicked on two images are collected and autoindexed with values returned for the space group and unit cell.

    Current developments, which will be tested on Station 7.2, involve an expert GUI that will be able to take the (editable) output from the strategy option.

  6. Progress at Cambridge
  7. Graeme reported that the mosflm server had been re-written to improve the code and he expected further progress should be made once the issues over XML had been resolved. Documents were being used to learn how to parse information automatically. Once parsing has been fully implemented it was expected that future developments would be easier. A separate description of the XML that the programs will use is required.

    The issue of asynchronous abort handling was discussed. Currently an "abort" request is sent through the same channel as all other requests. However the abort must be handled asynchronously. Currently when using the server, the server has to be asked to stop the mosflm process by connecting to the server. The server will then create a thread connecting to mosflm requesting it to stop. It was pointed out that currently one cannot stop mosflm, one has to kill it. In most cases an elegant abort is required which still leaves other processes standing. It was pointed out that aborting mosflm on the same port would cause problems – the python sending the request to the server wont allow a second request before a response has been received back from the first request so a second port would be required.

    During the discussion it was realised that there are many instances when an abort might be required so a list was made to try and cover the various instances when an abort might be required:

    LIST OF ABORTING SITUATIONS:

    It was felt that the ability to abort a process cleanly was vital before any system could be put onto one of the beamlines.

    A timeout system has been put into the server already. The timer is re-set after a response is sent back after each command.

    Harry Powell reported on progress with mosflm. The mosflm code now contains the ability to write to the server using sockets and XML. The latest version is Version 6.12, which will be released in the first full week of February.

  8. Issues arising out of recent progress
  9. The issue of communication between the various programs was discussed. Concerns were expressed that there would be difficulties trying to run things remotely. It was also felt that any requirement for authentication would cause difficulties. A request was made for a specification on how to communicate with the mosflm server and how to receive information back.

    The issue of alternative data processing programs was raised – other programs are used on the beamlines such as XDS, d*trek, HKL2000, denzo plus other strategy programs as well. Communication between the expert system and the data collection and data processing programs will require a translator module. The translator would take generic commands and translate them into something understood by each of the individual data processing programs. It was also pointed out that a version of the mosflm server plus data acquisition program plus expert system would ultimately be required for home sources. It was felt that it was the responsibility of the expert system to take care of the genericness required to cope with the many programs and not the mosflm server.

    The movement of the discussion towards the generic nature of the expert system brought the discussion around to the requirement for a definition of the project as a whole. It was felt that in order for progress to be made one should have a definition that related to mosflm but that it should be made as generic as possible. Communication with the mosflm server would be mosflm specific – the information would be tagged with mosflm tags. However if, for example XDS were used, then the information would be passed with XDS tags. This raised the issue of the ability to communicate with the various different programs available – contact would ultimately be required with the authors of the various programs. It was also pointed out that it would be useful to find out what was happening at the various synchrotron sources so that effort in this area was not duplicated. Communication in this area could be through the MAX-INF network. Currently the next MAX-INF meeting is expected to be in August at the Swiss Light Source.

  10. Version Handling
  11. It is vital to ensure that any changes are documented and that everyone knows whether they have the latest version or not. It is important to ensure that different versions are handled correctly and that there is sufficient change control in place so that any changes are known about and are well documented.

    It was felt to be sensible to have 2 version numbers (int.int) – one number to indicate the state of the interface and the other to indicate the state of the bug fixes in that particular version. There would need to be versions numbers for the mosflm server, the XML and mosflm itself. Similar requirements would have to be enforced for every other component of the expert system.

    It was suggested that CVS (Concurrent Version System) be used. This suggestion was taken on board and will be implemented 1 week after the XML definitions have been sorted out.

  12. Data Definitions – leading to XML
  13. Definitions are required for all aspects of the work in the DNA project. All of the developers should provide Steve Kinder with their XML definitions by Friday 8th February so that Steve Kinder can put the definitions on the DNA website. Harry should provide Steve Kinder with XML definitions as required for mosflm and the other should provide the definitions of the data and XML as they are passed between the data acquisition programs and the expert system.

    A DTD plus text description of the XML plus an example was felt to be sufficient information for an explanation of the interface.

  14. The Way Forward
  15. It was established that a revised version of XML would be put onto the web site by 22nd February.

    Following this a new version of the mosflm server incorporating the revised version of XML would be available on 1st March.

    Work at the ESRF and Daresbury would continue to develop the software demonstrated in the meeting so that it could be released onto the beamlines by the end of March.

    Currently Daresbury is using the mosflm server in single user mode. The ESRF are using it in multi-user mode governed by a root-owned daemon and require authentication before anything can be released onto the beamlines. Olof et al will discuss authentication issues with Graeme.

    The next stage of developments will involve returning suggestions for a strategy for data collection. Harry has already developed the strategy option within mosflm so that it will communicate with the server. Some further developments by Graeme are required before it can be fully released and made available for the beamline software.

    The strategy program written by Sasha Popov and collaborators at Hamburg was discussed. This program will return values for the phi rotation range required for data collection along with the D f required. From analysis of the image an exposure time required to give a mean I/s I is also suggested. The ability to collect in dose mode then modifies the exposure time during the course of data collection to ensure this mean I/s I is maintained. This program has been implemented on all of the beamlines at Hamburg and apparently works well as long as there is no radiation damage (the usual case at Hamburg). A request has been made to Victor Lamzin for the source code of the program, and a response has been received, offering to send binaries. This is a start and further discussions will take place with Victor.

    The Next Phase:

    Following on from returning suggest values for a strategy; the next stage would be to suggest a suitable value for the exposure time. As the project develops further one would expect all the boxes requiring numbers for data collection would be automatically filled by values given from the expert system. There should always be the ability for users to edit these values.

    Currently mosflm only sends back a single autoindexing solution, the query was raised – whether mosflm should return all solutions with the top one highlighted as default. The issue of what to do if it transpired that the top solution was incorrect was raised. It was pointed out that one would not need to stop the data collection, simply modify the data collection parameters and possibly re-process the data (depending on how far one had got with data processing).

    The possibility of using a web-based system was discussed. It was thought to be fairly easy to use http to carry out the required communication functions.

    The issue of mosflm keeping up with the data collection was raised again – this is an issue at the ESRF. It would be possible to split the mosflm job up so that for example the first 5 images were sent to 1 machine, the second 5 to another etc.

    The issue of data storage was raised. Increasingly protein crystallography beamlines have moved towards using databases. The use of XML tagged documents would also help in the ability to retrieve information. The query was raised as to what Stanford was doing in this area – is everything being recorded in a database?

    It was highlighted that it would be useful to find out if other SR sources are doing anything in this area. This is fairly easy to find out within Europe as there is the MAX-INF network – the next meeting of this is in August probably at the Swiss Light Source after the IUCr meeting. Colin said that he would inform other synchrotrons about what is going on, pointing them to the web site. It was suggested that this should only be done once useful screen shots of the demos/work done so far have been put onto the web site. Additionally more documentation of the project on the web site is also needed before directing people external to the project to the web site. Harry Powell is attending the ACA meeting in the USA and will prepare a poster on the DNA work for the meeting.

    NEXT DEVELOPERS MEETING: The next developers meeting will take place in mid-April at the ESRF. However if there is no beamtime available then to try things out then the meeting will take place in Cambridge. 24th April was suggested as a possible date.

    NEXT FULL MEETING: The next full meeting will take place in September either in Daresbury or Cambridge.

    ACTIONS SUMMARY: