Revolutionizing data access through new software tool: Tiled

Scientists can use Tied to seamlessly access databases in various formats such as files, databases or other databases. Tiled allows its users to view, cut and study their data using the most convenient tool for them. Credit: Brookhaven National Laboratory

Whenever scientists research new material for future batteries or research diseases to develop new drugs, they must march in an ocean of data. Today, an entire ecosystem of scientific tools creates a wild variety of data to be explored. This research will now become much easier thanks to scientists at National Synchrotron Light Source II (NSLS-II), located in the Brookhaven National Laboratory of the U.S. Department of Energy (DOE). Their new software tool – called Tiles– Allows researchers to see, cut and study their data more conveniently than ever before. This new data access tool makes finding and analyzing the right data for a walk in the park compared to previous methods, paving the way for the next scientific breakthrough.

As one of the 28 facilities of the DOE Office of Science across the country, NSLS-II welcomes nearly 2,000 scientists each year to use its ultra-bright light, while addressing the biggest challenges in materials and life sciences. These guest researchers come from around the world to collaborate with experts and use the special research tools in NSLS-II. They dump their specimens, ranging from ancient rocks to new quantum materials, with intense X-rays and capturing outgoing signals using advanced detectors. In turn, these detectors spit out streams of data, awaiting analysis by scientists.

“Working with data is a key part of any research, and yet a challenge in itself. It comes in a variety of formats, sizes and shapes, and not every part of it is useful for researchers. This is why developing software tools that make accessing, viewing and sorting data so important.” Said Dan Allen, a computational scientist at NSLS-II.

Tiled is a data access service for data-aware portals and data science tools. This means that Tiled sits on top of databases and file systems so that scientists can access their data through, for example, a web browser or data analysis software. While the Data Science and Systems Integration (DSSI) program was launched for Tiled for all NSLS-II test stations, the service, just like its cousin Bluesky (a data acquisition software also developed in NSLS-II), can be used in any research lab across the world. It is possible that Tiled is published under a popular open source software license.

“Although we developed Tiled in the Python programming language, so it integrates naturally with Python-based data libraries, nothing in the service is specific to Python,” said Stuart Campbell, chief data scientist at NSLS-II. “The client uses an API, or application programming interface, to connect the user applications to the server. An API is basically a set of rules, or a contract that defines how different pieces of software communicate with each other. The great thing about this approach is that once those rules and interfaces are defined, It provides users and developers with the structure within which they can build some great tools and expand functionality beyond what they originally envisioned. “

Tiled’s flexibility allows the service to integrate seamlessly with any database or file collection, so it can be used in a wide variety of experiments with very different techniques and data.

Spread your data needs squared

“In the past, I used to help my PhD consultant download data from facilities like NSLS-II. It was tedious because we had to download all our data at once before we could arrange the useful parts. In addition, the data was in detector format – regardless of how we wanted to analyze “It means that after a long download, we had to convert the data before we could even look at it,” Allen said.

Campbell added, “If Dan had been Tiled then, he could have easily browsed the data in a web browser or data analytics app, sorted out the good parts, and shared only the interesting ones with his advisor through a single link.”

A revolution in data access through Tiled

This preview of the Tiled Web Client shows how you can view different detector images of different sizes at the same time. The preview shows the portal in dark mode. Credit: Brookhaven National Laboratory

Using Tiled, scientists can preview their data and access only the parts they want without a major download. They can either select the format of the data you have downloaded or enter it directly into the analysis software. At the same time, Tiled offers access control based on Internet security standards so that all data remains secure. Because setting up a new account can be a barrier, Tiled can be configured to allow access to third-party services, such as Google and ORCID.

“Remote capabilities are more important than ever,” said Dylan McReynolds, a computer systems engineer at Advanced Light Source, a user facility of the DOE Office of Science located at Lawrence Berkeley National Laboratory that collaborated on Tiled. “Relying on open and standard Internet protocols advances our scientific capabilities by enabling data to be easily transferred to where it is needed.”

The new software even allows for a form of “flight mode” where the data is stored on the user’s laptop so that researchers can continue to work on it offline or in a slow internet connection.

“Our goal with Tiled is to simplify data access for everyone. If you do not have to worry about converting data formats to other formats or selecting information from file names, you can think of the more important parts, like finding the answer to your research questions,” he said. Said Thomas Caswell, a computational scientist at NSLS-II.

Simplifying and standardizing data access is critical to both optimizing existing workflows and enabling future workflows that focus on Machine Learning, AI, and other advanced analytics. These evolving technologies critically rely on a frictionless approach to data, regardless of how it is collected or stored, in order to exploit their full potential.

Tiles: Suitable for any research puzzle

Tiled’s first users have already built some exciting and sophisticated tools for running their research.

“Tiled offers an entirely new way to access data that will simplify and streamline processing and analysis lines for experiments. No more cumbersome downloads or wasted time importing data from a dozen experimental analysis formats!” Said Dennis Leshev, an assistant physicist at NSLS-II who tested Tiled. “In addition, Tiled will allow a simpler way to share data, paving the way for more open and transparent science in the future.”

The new software is not only available to NSLS-II users: the team has designed the software to be suitable for any data source. It can be deployed on a large scale for facilities like NSLS-II, but it can run equally well on a student’s laptop or on a research group workstation. Other laboratories and institutions already have the opportunity to adapt this software to their needs.

A revolution in data access through Tiled

This Jupyter notebook, a popular web analytics application, uses Tiled to access data for calculation, processing and imaging. Credit: Brookhaven National Laboratory

Peter Beaucage, a team scientist at the National Institute of Standards and Technology (NIST), who is an early user of Tiled, combined it with his own scientific data analysis program, PyHyperScattering. It enables Tiled to handle data transfer and security information, relying on it to provide its users with the specific interface they need for their work.

“The volume of synchrotron data needed for typical analysis has expanded dramatically in the last decade, expanding rapidly beyond the capabilities of existing data transfer platforms. Tile and similar solutions promise to give users seamless access to the right data at the right time and accelerate X-ray-based discovery,” Beaucage said.

Beyond Beaucage, other Tiled users have also built data analysis tubes, transferring data from live experiments in NSLS-II to remote clusters and custom software for viewing and exploring the data. Each step is supported by Tiled.

“Overall, we’re incredibly proud to distribute Tiled. It’s the culmination of our work over the last six years. It combines all the features we want in modern data access tools, and it goes hand in hand with Bluesky,” Campbell said.

The way forward

Tiles will allow a whole garden of useful tools to grow for a variety of techniques. The team is focusing on building different web applications that focus on specific research techniques. The team also wants to design a public data interface so that everyone can explore real data available to the public using Tiled.

“Grants often require open access to data, but researchers have a hard time getting it done in a practical and immediate way. Tiled puts a track to the researchers’ door, and works with the tools they already use to help them make data accessible, reciprocal, reusable and reusable, respectively. To FAIR’s guidelines for managing and managing scientific data, “Allen added.

By separating the way data is stored from access to them, Tiled opens up a way to use advanced storage and search technologies from within, while presenting researchers with time-tested and established standards. It meets them where they are and leaves them responsible for how to design and work with their data.

“Tiled strives to keep track of other NSLS-II software efforts in increasing a friendly community of donors and users. We are actively seeking collaboration with facilities and researchers around the world – whether in industry, academia or government – who have similar challenges, and we are excited to see what we can Build together on this platform, ”Allen said.

After AI mastered Go and Super Mario, scientists taught them how to ‘play’ experiments in NSLS-II

more information:
Daniel Allan et al, Bluesky’s Ahead: A Multi-Facility Collaboration for a la Carte Software Project for Acquisition and Management Data, Synchrotron Radiation News (2019). DOI: 10.1080 / 08940886.2019.1608121

Tile documentation:

Tile demonstration (for programmers):

Bluesky Open Source Project Home Page:

Provided by Brookhaven National Laboratory

quotation: Revolution in Data Access Using New Software Tool: Tiled (2021, November 24) Retrieved November 28, 2021 from

This document is subject to copyright. Apart from any fair transaction for the purpose of research or private research, no part may be reproduced without written approval. The content is provided for informational purposes only.




Please enter your comment!
Please enter your name here