Prepare observations for JEDI-SOCA exp (r2d2 database)

JEDI
1

Hi,
I am trying to run an experiments with my own observation database. My understanding is that first I have to convert the observation into ioda format, store them in a r2d2 database and link the r2d2 dir in the experiments. Would you please tell me where I can put the r2d2 dir? FYI, my r2d2_host is orion. In this case, do I need to worry about mysql which requires DatabaseCredentials such as username, password, host, Port etc.

2

Hello, mjhossen.

  • All R2D2 data store directories are now centrally controlled by the MySQL database.
  • R2D2 end users no longer create a separate directory for their data. A data store already exists on Orion for end user data.
  • R2D2 end users are required to have AWS permissions for the JCSDA-NOAA account on region “us-east-1”.
  • R2D2 pulls its MySQL credentials from an S3 bucket from this account and region.
  • R2D2 is integrated with a centralized, authoritative MySQL server running in “us-east-1”.

Here are the steps for ingesting your observations into R2D2.

(1) As you mentioned, you will convert your observations to the IODA format using the correct converter from GitHub - JCSDA-internal/ioda-converters: Various converters for getting obs data in and out of IODA.
(2) Export $R2D2_HOST="orion".
(3) Update your AWS config and credentials as per these instructions: https://github.com/JCSDA-internal/r2d2/blob/develop/TUTORIAL.md#aws-credentials. If you do not have access to the JCSDA-USAF account on “us-east-2”, then do not include that entry in each AWS file.
(4) Execute R2D2’s store call in the following manner with your values replacing provider, observation_type, file_extension, window_start, and window_length with your values for each source_file provided.

from r2d2 import R2D2Data
R2D2Data.store(
item=‘observation’,
provider=‘noaa’,
observation_type=‘amsua_n15’,
file_extension=‘nc’,
window_start=‘2020-01-01T00:00:00Z’,
window_length=‘PT6H’,
source_file=‘/path/to/ioda/file.nc’)

1 Like
3

I got error message related to AWS Access Key Id. I have no idea how I can get the AWS Access Key Id for Orion. Any suggestions for this?

4

The AWS access key is for your AWS user account for the JCSDA-NOAA accounts. It will be the same as the access keys you use locally on your laptop.

5

No success yet. I used my AWS personal account info. I do not have AWS for JCSDA-NOAA. I am wondering how I can link myself to JCSDA-NOAA.

6

Please contact the JEDI Infrastructure Team Lead at dom.heinzeller@ucar.edu. I do not have permissions to add users to that account.

7

I have run an existing experiments of JEDI-skylab as
create_experiment.py $JEDI_SRC/ewok/experiments/mom6-3dvar-5deg.yaml

I got an error message related to ewok/src/yamltools/template.py:
File “jedi-bundle/ewok/src/yamltools/template.py”, line 128, in substitute_template_variables
full_value = full_value.replace(f’$({g})', value)
TypeError: replace() argument 2 must be str, not None

Is it my mistake? Do I need to provide another yaml file? Or is it a bug?