Remote Data Collection

On This Page

Data Collection

The majority of our users perform data collection remotely from their home institution.

Remote Data Collection is available to all NE-CAT users through our custom browser-based general user interface (GUI). The GUI will work on most current web browsers but is designed to work in up to date versions of Firefox, Safari and Chrome. Microsoft Edge is not supported.

Guidelines

Prior to data collection, you will need to complete some tasks.  Check out our Pre-Data Collection guidance or the checklist below to make sure that you are ready to collect data.

  • Site Access – You must be a registered user and have site access (through My APS Portal).

  • ESH223U – You must be current on ESH223U training.

  • ESAF – All users operating the beamline/collecting data must be listed on the ESAF as type ‘Remote’.  The ESAF must be submitted 7 days before your beamtime.

  • Samples – Your dewar should be shipped to arrive at NE-CAT one working day in advance of your beamtime.

  • Sample Spreadsheet – You must supply a sample spreadsheet to your support staff for uploading into remote.

Sample Spreadsheet

Remote data collection is facilitated through use of a sample spreadsheet that allows for filenames to be assigned to samples.  The most current version of the spreadsheet is in .xlxs format and can be downloaded from the Remote website by clicking the Sample Spreadsheet link in the bottom left corner of the login screen.  Most of the entries on the spreadsheet are optional and are provided for you to keep track of your samples.  The current columns required by remote are: PuckID and CrystalID.

  1. PuckID should be the alphanumeric characters carved into the lid of your puck.  Staff will use this to assign your puck location in the robot dewar.
  2. PuckID needs to be filled out for every row that has a sample.
  3. CrystalID will be the filename to be associated with your crystalline sample.
  4. CrystalID should be alphanumeric characters.  The CrystalID will be used to name your snapshots and runs.  Each CrystalID in a spreadsheet should be unique.
  5. Any spaces or hyphens (-) in the PuckID and CrystalID will be replaced with underscores (_) when the spreadsheet is read into Remote.
  6. Each puck is a separate sheet in the workbook.  Do not rename the sheets which are called Puck1, Puck2, Puck3, etc.  Do not add spaces.
  7. Do not rename the columns or add extra column headings.  The spreadsheet will not read into remote if you rename the columns.  You do not need to fill out any columns except PuckID and CrystalID.
  8. SampleID comes prefilled on each sheet.  Do not delete any SampleIDs.
  9. Once you have filled out your sample spreadsheet, email it to your support scientist who will upload it to Remote.
  10. Turn on Auto-Naming in Remote.

Data Transfer

You can transfer your data either using the ‘sync’ script in Remote or using Globus.

Sync Script

If you collected your data via remote, you can download your data using the sync script.  Click ‘sync’ button next to the session you want to download on the homepage of the NE-CAT Remote GUI.  This will download the sync script to your machine.

You run the script from a terminal window on Linux or Apple systems using Python3; i.e.  ‘python .py’.  Running the command with no options will print a usage message, showing a set of options which you can use for downloading your data.  To download your data, run the script with your chosen options. This script runs rsync to transfer data to your filesystem, so if the transfer is interrupted you can simply restart the script in the same location and it will continue copying data that was not transferred.

You may download specific data files or from a target directory. However, if your rsync version is outdated, you will get a warning and this option will not work.  To upgrade rsync, install Homebrew from “https://brew.sh/”. Open terminal and run “brew install rsync”. Source to rsync from terminal “/opt/homebrew/bin/rsync”.

Use ‘-da’ (or ‘-dar’) to select specific files/directories (the directory tree may take a couple of seconds to load). Select single or multiple files (e.g., 6, 8, …) to download. Resume interrupted transfers by restarting the script. Use ‘-t’ to download from a specific directory.

The sync script that you download expires after 72 hours. If you still need to download data from a session after your download script has expired, simply return to the NE-CAT Remote GUI and download the sync script again, this resets the time limit.

Globus

NE-CAT now offers Globus for secure and reliable data transfer, in addition to the existing Python script.  Globus is a non-profit service for secure, reliable research data management developed and operated by University of Chicago (https://www.globus.org/our-story). You can move or share your data between two endpoints (file transfer locations) using the Globus interface. We have provided a set of instructions for setting up Globus and locating the source endpoint. We must send you an invite to use Globus.  If you need an invite or assistance, contact Ali Kaya.