A digital solution to the reproducibility crisis

The Journal of Irreproducible Results, which sadly no longer seems to be in print, was first published in 1955. Subtitled “Improbable Investigations & Unfounded Findings”, it entertained generations of scientists and would-be scientists with its satire of academic works; one of its later editors, Marc Abrahams, went on to found another celebration of dubious scientific endeavours, the Ig Nobel Prizes.

 

In the chemistry lab, however, irreproducible results are no laughing matter. Reactions that fail to work consistently cause frustration and waste time, energy and expensive reagents. Even when it seems that all variables have been carefully controlled, yields can vary hugely or reactions can fail altogether. In a 2016 Nature survey of this “reproducibility crisis” in science, almost 90% of chemists said they had failed to successfully repeat someone else’s experiment, and more than 60% had failed to reproduce their own results. Rick Danheiser, editor-in-chief of the journal Organic Syntheses and Arthur C. Cope Professor of Chemistry at the Massachusetts Institute of Technology, wrote in an article entitled “Reproducibility in Chemical Research” the same year: “Replicating results in synthetic organic, organometallic, and inorganic chemistry continues to present problems even for experienced and skilled researchers.”

 

This unfortunate reality of real-life chemistry often manifests when processes have to be transferred from team to team or location to location. In today’s world, where international travel may be limited for some time to come, and process handovers are increasingly virtual rather than “hands-on”, the problem of irreproducibility might only worsen. The “green-fingered chemist” who can get any reaction to work while colleagues fail may not be on hand to apply their magic touch.

 

DeepMatter, a chemistry technology company with teams in both Glasgow and Munich, is tackling the reproducibility crisis with DigitalGlassware, its innovative cloud-based digital chemistry platform. The technology enables the capture of real time data during a reaction including from its packaged array of sensors which can be placed inside a reaction vessel, along with ambient environmental sensors, which are linked via the cloud to integrated reaction structuring, planning and analysis software. When carrying out a reaction using DigitalGlassware, the chemist follows a “recipe” that is recorded in a standardised format. Each step of the reaction is recorded and timestamped, while the data from the sensors (including temperature, pH, UV transmissivity, etc) is automatically recorded and correlated against the actions taken in the lab. In this way, the effects of each step carried out by the chemist can be seen immediately by others monitoring the reaction remotely, and can be analysed in detail after the event.

 

Individual repetitions of a reaction can be compared side by side, making it simple to highlight any differences or departures from the published method, for example deviations of temperature, quantity of reagents, or timings. These findings can then be used to optimise the reaction recipe, which is easily transmitted to other chemists for future work.

 1 https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

 2 https://onlinelibrary.wiley.com/doi/full/10.1002/anie.201606591

To investigate the utility of DigitalGlassware in improving the reproducibility of a reaction, DeepMatter recently undertook a study with a commercial discovery chemistry firm. The aim was to take an industrially relevant reaction, and show how the integrated DigitalGlassware platform could define the method in a standardised, shareable format, then capture and analyse data allowing the reaction to be modified and optimised.

 

The reaction chosen for the study was a Buchwald–Hartwig amination (Fig. 1), which synthesises C-N bonds via coupling of an amine with an aryl halide. Since being described independently by Stephen Buchwald and John Hartwig in 1994, this method has gained wide use in synthetic organic chemistry, with applications in many total syntheses and the industrial preparation of numerous pharmaceutical molecules. The metal-catalysed process has several advantages, chiefly that it allows relatively mild reaction conditions and has a wide scope, allowing virtually any amine to be coupled with a wide variety of aryl coupling partners. The particular cross-coupling reaction chosen for this study uses a palladium catalyst with the phosphine ligand MePhos.

 

To assess the performance of DigitalGlassware, the reaction was deliberately de-optimised. The published scheme uses tert-amyl alcohol as the solvent (boiling point 103 ºC); this was replaced with isopropyl alcohol, which is inexpensive and boils at a lower temperature (83 ºC). It was anticipated that the change of solvent would reduce the yield when the reaction was initially run; the intent was to replicate the early stages of reaction development and allow the method to be optimised in a digital framework.

 

Protocol

Phase 1: Traditional lab notebook method

Initially, an experienced chemist carried out the reaction in the traditional way, without the assistance of DigitalGlassware, and recorded the method in a laboratory notebook. The solution was sampled every 30 minutes and analysed using high-performance liquid chromatography (HPLC), to provide a baseline with which to compare future results. This initial reaction was performed three times.

Phase 2: Encoding with DigitalGlassware 

The chemist then transferred the method into the standard DigitalGlassware “recipe” format. The recipe files can be shared instantly across users of the DigitalGlassware platform, wherever they are located; similarly, modifications can be made and transmitted immediately on the cloud-based system.

 

Phase 3: Following DigitalGlassware recipe

The same chemist now used the DigitalGlassware platform to repeat the reaction in triplicate. This involved, firstly, following simple step-by-step instructions on the lab-based RecipeRunner tablet app to set up the apparatus, including installing the self-contained DeviceX sensor array inside the reaction vessel and the EnvironmentalSensor to record ambient conditions. Once the reaction was set up, the chemist followed the DigitalGlassware recipe step by step, recording progress in the app, while the sensors collected data and transmitted it to the cloud in real time. Because DigitalGlassware timestamps both reaction steps and sensor data, results can easily be correlated at any stage of the process, and reviewed and compared using the RecipeRunner component of the platform. Again, samples were collected and analysed using HPLC.



Results

The two chief findings of this study were that the average reaction yield was significantly increased, and the variability greatly decreased, when using DigitalGlassware, compared with traditional methods. Directly comparing the reaction runs using the two methods, the yield was 50% greater when using DigitalGlassware, with the average error reduced by 80%.

After two hours of reaction time, the measured conversion without DigitalGlassware was 26%, plus or minus 24%. At the same timestamp when using DigitalGlassware, the yield was 39%, plus or minus 2%, demonstrating an enormous increase in reproducibility.

 

Analysis and optimisation opportunities

The DigitalGlassware platform incorporates analysis software which allows detailed comparison of reaction runs, using the real-time data recorded and transmitted by the sensors, timestamped alongside each recipe step, and other data such as photographs taken in the lab. This time-course data collection approach adds vital context, enabling chemists to identify areas for reaction optimisation. To take a simple example, the HPLC analysis of the reaction being studied showed that the conversion proceeded very little after 3 h of reaction time. Continuing the reaction beyond this point produces little additional gain but consumes energy and takes up valuable time which could be spent more productively. With staff and overhead costs estimated to be at least $350,000 per annum per full-time chemist in large pharmaceutical firms, even a modest increase in productivity represents considerable financial gain.

 

With a traditional experimental setup, assessing the completion point would require costly and labour-intensive analysis at regular periods. With the DeviceX in-flask sensor, changes in the reaction vessel are monitored in real time. By correlating these changes with the quantity of product present, the optimum reaction time and conditions can be assessed, and used to fine-tune the DigitalGlassware recipe file. 

 

Such information can also add to the chemistry understanding in a more qualitative manner. To take one example, relative humidity is an important consideration when carrying out moisture-sensitive chemistry, and easy measurement of this within the fumehood enables a chemist to quickly assess whether additional precautions may be appropriate and to compare to other similar experiments that may have been run at remote facilities.

 

The richness of data collected by DigitalGlassware opens the way for much more data science. Algorithmic methods are capable of identifying trends in data that may be missed by the laboratory chemist. Analysis of this rich data set may provide parameters that can be directly correlated with experiment outcomes: a trivial example would be be a simple colour change While this may be readily visible to the naked eye, the sensors will enable a more quantitative interpretation which can be better correlated with reaction outcomes, particularly given their separate RGB channels. This offers the opportunity to use these to monitor a reaction in real time and reduce the dependence on higher-cost analytical methods.

 

Summary

Mark Warne, CEO of DeepMatter Group, believes the advantages of DigitalGlassware will only become greater as the chemical and pharmaceutical industries adapt to the post-Covid-19 world. “With the increase of remote working and limits placed on travel between sites, it is increasingly likely that the valuable ‘green-fingered’ chemist won’t be available in person to share their expertise,” he says. “DigitalGlassware not only allows chemists to keep track of their own reactions, it permits that knowledge to be transferred seamlessly across organisations, ensuring that the most efficient methods and processes are being followed by every team member, no matter where they are located.” 

 

By creating a structured, standardised recipe rather than ad hoc notes in a lab book, DigitalGlassware will allow even inexperienced chemists to improve consistency and efficiency - and help to consign those irreproducible results to the pages of history.