Improving productivity with aggregated data using DigitalGlassware®

Reproducibility in the chemistry lab is an issue that most chemists have experienced in their careers and wastes not only time and money, but can be infuriating when the reasons aren’t clear! DigitalGlassware® from Deepmatter enables a higher level of experimentation through the structuring, collection and collation of relevant chemical data, creating rich run records that leave irreproducibility a thing of the past.

Read on to find out more about how DigitalGlassware® can accelerate the digitisation of your data collection in the chemistry lab and lead to improvements in productivity and efficiency.

Introduction

The reproducibility crisis

Reproducibility in the chemistry laboratory, specifically the ability to reproduce a chemical synthesis procedure as written [1], is an issue that is widely recognised within the life sciences community [2-6]. Several Nature studies indicate that as many as 50% of chemists have failed to reproduce a protocol from the literature [2] whilst as many as 89% of respondents believe there is a ‘crisis of reproducibility’ in their field [3]. It has also been proposed that chemists can spend between 50-75% of their lab time attempting to reproduce a colleagues protocol [4]! The journal Organic Syntheses, unique in it’s acceptance process in that every submission and experimental result therein must be reproduced independently by a member of the board of editors before publication, indicated that between 2010 - 2016 1 in every 13 articles submitted were rejected due to irreproducibility issues, even with assistance of the original authors in place [5]

But the issue of irreproducibility also has significant wider implications. Previous estimates suggest irreproducibility in life sciences costs $28bn annually in the US alone [6], and although progress has been made in bringing sustainable practices to chemical waste management strategies, the EPA reported non-recoverable chemical waste disposal in the US during 2013 reached 1.8mn tonnes [7]. Tackling irreproducibility will not only have an economic impact, but it will increase efficiency in the lab and provide more sustainable lab practices through reduced chemical wastage.

The root cause

So what are the causes of irreproducibility in chemistry? A breakdown of contributing factors in one Nature survey indicates a consistent sentiment within the community: a lack of data transparency [3]. Selective reporting of results, insufficient information within the experimental protocol, and even lack of access to raw data were all suggested to play a role in the inability to reproduce reported results. But the experience of the Organic Syntheses board of editors would suggest that data transparency means more than just providing additional information within an experimental protocol. There is a need to provide a structure and contextualisation to what the chemist does in the laboratory to facilitate a deeper understanding of the chemistry that has occurred and capture the root causes of variability in experimental outcome.

This is where DigitalGlassware® offers the solution.

DigitalGlassware®

DigitalGlassware® is a chemistry digitisation platform that provides a way to capture and collate data from the chemistry lab to provide infallible run records for improved reproducibility, analysis and interpretation. We combine structured protocols with time series data and end point outcomes to create rich run records that can be viewed through our web browser interface or downloaded for further analysis. By leveraging cloud-based technologies DigitalGlassware® facilitates access to aggregated data and simplifies the dissemination of results with your colleagues, no matter where you are in the world. In addition the structured protocols that underpin the data collection platform can be leveraged for advanced data analysis and ML models, providing opportunities for improved insights and understanding.

In this white paper we will explore how DigitalGlassware® enables improvements in chemical understanding through the capture and collation of time course data, including in situ sensor data, contextualised with the chemists actions in the lab, notes and observations when they occur and end point outcomes. Importantly this white paper will explore the results captured across multiple chemistry sites that are centrally connected through the DigitalGlassware® cloud based platform, enabling access to data in real time from a secure web browser.

This white paper highlights:

  • Enabling data digitisation strategies through the implementation of the DigitalGlassware® platform

  • Correlation of chemists actions with outcomes through the passive capture of time course sensor data

  • Ease of accessibility of rich data streams from multiple labs through the DigitalGlassware® cloud based platform

Read on to find out more about how DigitalGlassware® can enable digitisation of your lab to make your workflow better, faster and cheaper.

Choosing the chemistry

The procedure chosen was the “Large Scale Synthesis of Enantiomerically Pure (S)-3-(4-Bromophenyl)butanoic Acid” reported by Ruble et al in Organic Syntheses [8]

The procedure chosen was the “Large Scale Synthesis of Enantiomerically Pure (S)-3-(4-Bromophenyl)butanoic Acid” reported by Ruble et al in Organic Syntheses [8]

For this study a Michael addition [8] was chosen for several reasons: not only is it an important carbon-carbon bond formation reaction but this particular synthesis has numerous aspects that can influence reproducibility, and ultimately productivity, requiring a level of skill from the chemist. For instance there is an air sensitive component to this synthesis that demands care, a distinct colour change occurs (from yellow during catalyst formation to a strong red colour) and importantly the catalytic activity of the transition metal is highly dependent on the purity of the batch. Multiple factors need to be considered for this synthesis to ensure success, and is an ideal candidate for this multi-site assessment.

The project scope

The scope of the project was to provide two sets of chemists, who have never met nor communicated before, with identical experimental procedures and for them to perform the synthesis using the DigitalGlassware® platform. By doing so we can assess approaches from the different chemists, how this leads to variability and how we can learn and develop the experimental procedure from the aggregated data. With the chemists using DigitalGlassware® we can ensure all their actions can be tied closely to the time course data streams the platform captures and collates [Rich data capture with DeviceX from DigitalGlassware® | DeepMatter ]. Additionally both sets of chemists had access to the same DigitalGlassware® Cloud based platform, meaning all their data is automatically collated and accessible in the same place, irrespective of where they were doing their chemistry [DigitalGlassware® : Different Locations, Reproducible Results | DeepMatter].

The only requirements we set on the chemist were are as follows:

  • Each chemist must run the same DigitalGlassware® Recipe in triplicate

  • DigitalGlassware® hardware (our DeviceX and EnvironmentalSensor) should be used for passive monitoring of the time course data

  • The chemists should capture their experimental set ups using the DigitalGlassware® RecipeRunner application when they begin their experimental runs [Removing ambiguity in the lab with DigitalGlassware® | DeepMatter ]

Getting started

The first step in the study was converting the experimental protocol into a DigitalGlassware® Recipe. This was performed quickly and easily using the RecipeBuilder module in our Web App service and allows the chemist to coordinate their actions in the lab closely with the capture of the time course sensor data [Working within your lab workflow with DigitalGlassware® | DeepMatter ]. Once the Recipe was in place it was shared instantly through the DigitalGlassware® Web App and made available for the chemists to access and perform.

The procedure was performed in triplicate at two different sites involving two different chemists (Chemist A and Chemist B). Each chemist ran the same Recipe in DigitalGlassware® based on the Organic Syntheses procedure but was free to explore and prepare their experimental set up. Real time monitoring was achieved using our DigitalGlassware® kit (including a DeviceX and EnvironmentalSensor), allowing us to capture variations in situ including temperature and stir-rate, and to monitor ambient lab conditions. Post run analysis was performed using a combination of LC/MS and 1H NMR and was associated with each DigitalGlassware® run record. Additionally our RecipeRunner Android app offered opportunities for the chemists to record their observations, an incredibly useful feature to capture and correlate equipment set ups and observed colour changes across the different sites.

All data has been collated on our DigitalGlassware® Cloud platform, allowing simple access to anaylse compare and share data from your web browser irrespective of your location. For this study the data captured at two different sites was analysed and interpreted at a third site: the Deepmatter offices.

Results

Similar approaches…

As the chemists were free to explore their experimental set ups for the reactions, it was interesting to observe how different approaches could potentially affect reproducibility and outcomes. Although a chemists experience and training will determine adopted best practices, availability of hardware can also influence and the capabilities to perform specific chemistry. In the below figure we can see subtle variations in the set up both chemists adopted: one used a traditional oil bath with a stop cock connector for gas control whereas the other used a heating block and opted for a rubber septum for gaseous control. Note that both these images were taken using the DigitalGlassware® RecipeRunner at the start of the Recipe run.

A comparison of the experimental approaches both chemists took for their planned chemistry

A comparison of the experimental approaches both chemists took for their planned chemistry

…but different results?

The outcomes of the reactions demonstrate a wide degree of success: only two of the six runs reported a reasonable yield (about 45%) whilst the remaining runs reported yield values below 5%, and in some cases the product couldn’t be isolated. When discussing the results with the chemists they were confident that the procedure they performed was identical in each iteration, so what is the cause of the variation? Fortunately through the time course sensor capture with DigitalGlassware® we can dive deeper into the data to find out…

 

Run #

Total Synthesis Duration

Final Yield (%)

Purity (%)

Chemist A

1

23 hours 19 minutes

0

0

2

22 hours 35 minutes

48.5

52

3

23 hours 27 minutes

47.3

62

Chemist B

4

18 hours 29 minutes

3.5

96

5

19 hours 56 minutes

4.0

69

6

18 hours 38 minutes

0

0

Initial analysis of the time course sensor data revealed quite similar trend behaviour across all six runs, even between the highest and lowest yielding reaction runs. The following plots were taken directly from the DigitalGlassware® RunManager module that provides a quick and simple way to visualise and compare captured run data.

DigitalGlassware® provides a mechanism to correlate the chemist’s actions in the lab with time course data streams, enabling contextualisation of the trends and events during the reaction with what the chemist has performed. In the above example we are looking at the early temperature trend for Run 2. Note that in the example above the chemist’s have captured time stamped notes and observations against their data using the RecipeRunner tablet application.

DigitalGlassware® provides a mechanism to correlate the chemist’s actions in the lab with time course data streams, enabling contextualisation of the trends and events during the reaction with what the chemist has performed. In the above example we are looking at the early temperature trend for Run 2. Note that in the example above the chemist’s have captured time stamped notes and observations against their data using the RecipeRunner tablet application.

Similar reaction sensor profiles were obtained from all the runs, including those for the worst recorded outcome (Run 1, 0%, orange line above) and the best recorded outcome (Run 2, 48.5%, purple line above). Identified along the time course data streams are several actions that have been defined in the DigitalGlassware® Recipe. These actions, or operations as they are called in DigitalGlassware®, have been time stamped against the sensor data when the chemist has performed and completed the operation. This allows us to correlate all the incoming time course with that operation. For example the addition of 1,4-dioxane leads to a drop in temperature whilst the addition of triethylamine leads to a very small exotherm.

Similar reaction sensor profiles were obtained from all the runs, including those for the worst recorded outcome (Run 1, 0%, orange line above) and the best recorded outcome (Run 2, 48.5%, purple line above). Identified along the time course data streams are several actions that have been defined in the DigitalGlassware® Recipe. These actions, or operations as they are called in DigitalGlassware®, have been time stamped against the sensor data when the chemist has performed and completed the operation. This allows us to correlate all the incoming time course with that operation. For example the addition of 1,4-dioxane leads to a drop in temperature whilst the addition of triethylamine leads to a very small exotherm.

All the runs captured by Chemist A and Chemist B indicated a correlation between various time course sensor streams. In the above example we observe a drop in the temperature and UV-A light intensity upon the addition of 1,4-dioxane, suggesting the transmissibility of the reaction solution drops after this event…

All the runs captured by Chemist A and Chemist B indicated a correlation between various time course sensor streams. In the above example we observe a drop in the temperature and UV-A light intensity upon the addition of 1,4-dioxane, suggesting the transmissibility of the reaction solution drops after this event…

…however the drop in temperature could be correlated with a rise in agitation that would increase the dispersion of the 1,4-dioxane in the solution and therefore increase the surface coverage of the DeviceX probe in the reaction solution.

…however the drop in temperature could be correlated with a rise in agitation that would increase the dispersion of the 1,4-dioxane in the solution and therefore increase the surface coverage of the DeviceX probe in the reaction solution.

Capturing the colour change

All runs reported colour changes during the course of the reaction:

  • Upon the addition of dioxane to form the catalyst for the reaction (clear to yellow)

  • After the addition of water (yellow to dark red).

  • A lightening of the reaction solution with time (dark red to a lighter red)

Colour change is an often subjective aspect of reaction observation and can lead to ambiguity in terms of description but DigitalGlassware® can provide objective capture of these ambiguous events, providing improved opportunities for comparison and analysis. Both Chemist A and Chemist B made use of the DigitalGlassware® RecipeRunner app to capture these colour change events using the photo note feature. This allows us to visually assess variation in the reaction solution across the different reaction runs. Importantly however the chemists were able to make use of DigitalGlassware®’s sensor capabilities to record in situ events using the DeviceX camera. In doing so we can remove ambiguity in colour assignment and speak directly to variations in colour and turbidity based on real time course data.

Each group of chemists recorded colour changes using the photo note feature of RecipeRunner, timestamping observations against the DigitalGlassware® Recipe and the time course sensor data captured during the run. Here we can observe slight variations in colour upon the addition of dioxane, particularly with Run 2 and Run 3, the most successful reactions based on yield. However the addition of water produces a consistent deep red colour.

Each group of chemists recorded colour changes using the photo note feature of RecipeRunner, timestamping observations against the DigitalGlassware® Recipe and the time course sensor data captured during the run. Here we can observe slight variations in colour upon the addition of dioxane, particularly with Run 2 and Run 3, the most successful reactions based on yield. However the addition of water produces a consistent deep red colour.

An in situ camera streaming data to the DigitalGlassware® cloud was able to correlate fluctuations in turbidity and solution colour with the actions of the chemist. Here we have a comparison of the red channel value from the camera for the most successful runs (Run 2, 48.5% yield and Run 3, 47.3% yield) with the least successful runs (Run 1 and Run 6, 0% yield each) during the addition of ethyl (E)-but-2-enoate. Interestingly before the addition of ethyl (E)-but-2-enoate all runs indicate similar values in the red channel which corresponds to the observed red colour from the photo notes. Upon addition of ethyl (E)-but-2-enoate the solution begins to darken leading to an attenuation in all of the RGB values from the camera. During the remainder of the reaction the solution begins to lighten again. But two important events can be observed in comparison of the most and least successful runs: 1. Run 1 begins to lighten several hours before the more successful Run 2 and Run 3, and 2. Run 6 never recovers to the same level as the more successful reactions. Could both of these events be indicators towards success?

An in situ camera streaming data to the DigitalGlassware® cloud was able to correlate fluctuations in turbidity and solution colour with the actions of the chemist. Here we have a comparison of the red channel value from the camera for the most successful runs (Run 2, 48.5% yield and Run 3, 47.3% yield) with the least successful runs (Run 1 and Run 6, 0% yield each) during the addition of ethyl (E)-but-2-enoate. Interestingly before the addition of ethyl (E)-but-2-enoate all runs indicate similar values in the red channel which corresponds to the observed red colour from the photo notes. Upon addition of ethyl (E)-but-2-enoate the solution begins to darken leading to an attenuation in all of the RGB values from the camera. During the remainder of the reaction the solution begins to lighten again. But two important events can be observed in comparison of the most and least successful runs: 1. Run 1 begins to lighten several hours before the more successful Run 2 and Run 3, and 2. Run 6 never recovers to the same level as the more successful reactions. Could both of these events be indicators towards success?

Trusting your equipment!

The success of a chemical reaction is often (and rightfully!) lands at the feet of the chemist performing the chemistry. But a chemist is only as good as the tools they have at their disposal. As we know from experience a lot of faith is placed in the quality of chemicals and equipment used during the synthesis, and if this quality isn’t up to par then the chemists job is made increasingly difficult irrespective of skill and capability. A critical part of this synthesis was the temperature maintenance at 30 oC for 21 hours, and assuming the chemist works a normal working day, most of the synthesis will be left unattended overnight. Therefore how can we be sure the reaction will proceed as expected? Fortunately DigitalGlassware®’s passive collection of data removes doubt as to what happened when the reaction is left to it’s own devices.

One of the interesting observations from comparing the experimental set ups of Chemist A and Chemist B was the differing approaches to heating. Through DigitalGlassware® we can see quite clearly the impact this has on the reaction. The above figure compares the temperature trace from the most successful run Chemist A performed (Run 2, purple) that utilised the heating mantle approach, compared with the most successful run Chemist B performed (Run 5 orange) that utilised the oil bath approach. Note the fluctuation in temperature with the oil bath when active.

One of the interesting observations from comparing the experimental set ups of Chemist A and Chemist B was the differing approaches to heating. Through DigitalGlassware® we can see quite clearly the impact this has on the reaction. The above figure compares the temperature trace from the most successful run Chemist A performed (Run 2, purple) that utilised the heating mantle approach, compared with the most successful run Chemist B performed (Run 5 orange) that utilised the oil bath approach. Note the fluctuation in temperature with the oil bath when active.

Analysing the time course sensor data

We have so far surveyed the data as it has been presented to us, but what else can we gleam from the data, and what other potential correlates could be present that indicate why Run 2 and Run 3 by Chemist A was so successful? All DigitalGlassware® run data is available for download in the aggregate either by our XML record, PCRR [DigitalGlassware®: Get more out of your chemistry data | DeepMatter ], or by CSV, enabling export of the rich run records captured using the platform for other analysis applications. Here we performed simple statistical analysis calculating the Pearson correlation coefficient of the yield against a variety of variables derived from the in situ sensor streams and operation durations.

The highest correlation we were able to extract from the data sets collected related to the addition time of ethyl (E)-but-2-enoate. Interestingly the original procedure posited the addition time should be controlled over 5 minutes. However both Chemist A and B were unable to breach a yield of 10% when the addition time was below a 5 minute period. Chemist A was able to achieve a degree of success when extending the addition time past 5 minutes. Perhaps more importantly is Chemist A was unaware the addition time took so long! With DigitalGlassware®’s passive data capture we were able to capture this subtle but important deviation.

The highest correlation we were able to extract from the data sets collected related to the addition time of ethyl (E)-but-2-enoate. Interestingly the original procedure posited the addition time should be controlled over 5 minutes. However both Chemist A and B were unable to breach a yield of 10% when the addition time was below a 5 minute period. Chemist A was able to achieve a degree of success when extending the addition time past 5 minutes. Perhaps more importantly is Chemist A was unaware the addition time took so long! With DigitalGlassware®’s passive data capture we were able to capture this subtle but important deviation.

Summary and Learnings

The study described in this white paper demonstrates the power of having access to the DigitalGlassware® platform, not only in terms of routine reaction monitoring but the collation of data in a single place to facilitate analysis and tangible improvements to productivity in the lab. The variation in approaches to experimentation by Chemist A and Chemist B along with the time course data has been brought together in the DigitalGlassware® platform to provide an infallible and rich reaction record. By collecting this data we have been able to identify several areas of focus and improvement for both chemists for when this synthesis is performed:

  • Addition time of ethyl (E)-but-2-enoate is critical to the successful outcome of the reaction

  • Experimental approaches (and the choice of equipment) play a key role in developing confidence that the mechanisms in place for controlling applied condition variables (such as temperature and stir rate) are behaving as expected, and shouldn’t be taken for granted

  • Observable metrics such as colour change are powerful indicators for success, while unambiguous labels leave room for interpretation

  • Access to a centralised, aggregated data store expedites reaction trouble shooting and will facilitate design of experiment approaches and productivity improvements

DigitalGlassware®: a route towards a higher level of experimentation

Through the application of DigitalGlassware® the chemist can focus on what’s important: the chemistry. The passive data collection DigitalGlassware® offers enables a higher level of experimentation through the structuring, collection and collation of time course data from the reaction flask, and the capture of end point outcomes and observations. By integrating DigitalGlassware® with lab instruments it closes the gap in traceability, transparency and cost/ time efficiency during the reaction. The ability to provide aggregated data accessibility facilitates improved analysis of reaction data to ensure when something goes wrong, it was captured and is accessible instantly.

DigitalGlassware® can accelerate digitisation strategies in any chemical laboratory. The simplicity in its installation and application means organisations can start building data sets that will enable productivity improvements and insights from today.

To find out more about how DigitalGlassware® can help you with your data collection, or to request a free demo of the platform, click the button below.

References

  1. Reproducibility and Replicability in Science.; National Academies of Sciences, Engineering, and Medicine.; The National Academies Press, 2019; pp 42-46. DOI: 10.17226/25303.

  2. Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 2016, 533, 452-454. DOI: 10.1038/533452a

  3. Springer Nature, Author views, Researchers and Audience (Oct’ 2017). Nature journal ‘reproducibility’ survey. Nature Research, 2017. https://media.nature.com/original/magazine-assets/d41586-018-04590-7/15675426 (accessed 2022-03-31).

  4. Nathan Collins talks about the SynFini™ automated chemistry platform (Part 1). SRI International. YouTube, May 21, 2020. Nathan Collins talks about the SynFini™ automated chemistry platform (Part 1) (accessed 2022-03-31).

  5. Bergman, R. G.; Danheiser, R. L. Reproducibility in Chemical Research. Angew. Chem. Int. Ed. 2016, 55, 12548-12549. DOI: 10.1002/anie.201606591.

  6. Freedman, L. P.; Cockburn, I. M.; Simcoe, T. S. The Economics of Reproducibility in Preclinical Research. PLoS Biol. 2015, 13, 1-9. DOI: 10.1371/journal.pbio.1002165.

  7. Ritter, S. K. EPA Analysis Suggests Green Success. C&EN, February 2, 2015, updated February 2, 2015. https://cen.acs.org/articles/93/i5/EPA-Analysis-Suggests-Green-Success.html (accessed 2022-03-31).

  8. Ruble, J. C.; Vandeveer, H. G.; Navarro, A. Large Scale Synthesis of Enantiomerically Pure (S)-3-(4-Bromophenyl)butanoic Acid. Org. Synth. 2018, 95, 328-344. DOI: 10.15227/orgsyn.095.0328.