Hello, I have done the Tutorial: Simulating Observations with a JEDI-MPAS Application. If I want to do the observation simulation with other data, where can I get these data(nc4)? And is my main work to modify the hofx.yaml? Thanks!
The observation processing for mpas-jedi is relatively immature. There is no archive of IODA files ready for general use.
To try observations from a different period, or different observations (e.g. radiances from other instruments), you’ll need to generate new IODA files. What we have done so far is use the converter https://github.com/JCSDA-internal/ioda-converters/tree/develop/src/ncar-bufr2nc-fortran to convert observation files from NOAA/EMC that are archived on the NCAR RDA. (See my next post for links to the specific data sets.)
Observations from new periods or new observation types will require modifying the yaml’s supplied in the tutorial.
NCAR RDA data sets: https://rda.ucar.edu/datasets/ds735.0/, https://rda.ucar.edu/datasets/ds337.0/
As I understand it, this means that I need to use the bufr2nc_fortran.x. When I tried to compile the ioda-converters, I encountered the following error:
Singularity> ecbuild -DBUILD_IODA_CONVERTERS=ON ..
Found CMake version 3.20.0
cmake -DCMAKE_MODULE_PATH=/usr/local/share/ecbuild/cmake -DBUILD_IODA_CONVERTERS=ON ..
-- The CXX compiler identification is GNU 9.3.0
-- The Fortran compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/mpicxx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/local/bin/mpifort - skipped
-- Checking whether /usr/local/bin/mpifort supports Fortran 90
-- Checking whether /usr/local/bin/mpifort supports Fortran 90 - yes
-- ecbuild 3.6.1 /usr/local/share/ecbuild/cmake
-- cmake 3.20.0 /usr/local/bin/cmake
-- ---------------------------------------------------------
-- Found Git: /usr/bin/git (found version "2.25.1")
-- The C compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test iodaconv_C_FLAG_TEST_1
-- Performing Test iodaconv_C_FLAG_TEST_1 - Success
-- Added C flag [-pipe]
-- Performing Test iodaconv_CXX_FLAG_TEST_1
-- Performing Test iodaconv_CXX_FLAG_TEST_1 - Success
-- Added CXX flag [-pipe]
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Check size of off_t
-- Check size of off_t - done
Problem opening /proc/cpuinfo
Problem opening /proc/meminfo
-- ---------------------------------------------------------
-- ---------------------------------------------------------
-- [iodaconv] (0.0.1)
-- Feature TESTS enabled
-- Found Python3: /usr/bin/python3.8 (found version "3.8.5") found components: Interpreter Development Development.Module Development.Embed
-- Skipping python tests that import pygrib
Problem opening /proc/cpuinfo
Problem opening /proc/meminfo
-- Found MPI_C: /usr/local/bin/mpicc (found version "3.1")
-- Found MPI_CXX: /usr/local/bin/mpicxx (found version "3.1")
-- Found MPI_Fortran: /usr/local/bin/mpifort (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found bufr: /usr/local/lib/libbufr_4_DA.a (found version "11.5.0")
-- Found: bufr
-- Found: NetCDF::NetCDF_Fortran
-- Enabled Component: gnssro
-- NOT-Found: oops - BUFR converter disabled
-- NOT-Found: ioda - BUFR converter disabled
-- Disabled Component: bufr
-- Disabled Component: satbias converter
-- Enabled Component: obserror converter
-- Found: bufr
-- Found: NetCDF::NetCDF_Fortran
-- Enabled Component: pbfortran
-- ---------------------------------------------------------
-- Build summary
-- ---------------------------------------------------------
-- system : [th-hpc4-ln0] [Linux-4.18.0-305.12.1.el8_4.x86_64] [linux.64]
-- processor : [x86_64]
-- endiness : Little Endian -- IEEE []
-- build type : [RelWithDebInfo]
-- timestamp : [20211207220244]
-- install prefix : [/usr/local]
-- bin dir : [/usr/local/bin]
-- lib dir : [/usr/local/lib]
-- include dir : [/usr/local/include]
-- data dir : [/usr/local/share/iodaconv]
-- cmake dir : [/usr/local/lib/cmake/iodaconv]
-- ---------------------------------------------------------
-- C -- GNU 9.3.0
-- compiler : /usr/local/bin/mpicc
-- flags : -pipe -O2 -g -DNDEBUG
-- link flags :
-- CXX -- GNU 9.3.0
-- compiler : /usr/local/bin/mpicxx
-- flags : -pipe -g -Wall -Wno-deprecated-declarations -O2 -g -DNDEBUG
-- link flags :
-- Fortran -- GNU 9.3.0
-- compiler : /usr/local/bin/mpifort
-- flags : -g -fbacktrace -O2 -g -DNDEBUG
-- link flags :
-- linker : /usr/bin/ld
-- ar : /usr/bin/ar
-- ranlib : /usr/bin/ranlib
-- link flags
-- executable [ -Wl,--disable-new-dtags ]
-- shared lib [ -Wl,--disable-new-dtags ]
-- static lib [ -Wl,--disable-new-dtags ]
-- install rpath : $ORIGIN/../lib
-- common definitions: NDEBUG
-- ---------------------------------------------------------
-- Feature summary
-- ---------------------------------------------------------
-- The following features have been enabled:
* TESTS, iodaconv: Enable the unit tests
-- The following OPTIONAL packages have been found:
* Git
* Eigen3
* eckit (required version >= 1.13.0)
* gsl-lite
* MPI
* bufr
* NetCDF
* Perl
-- The following REQUIRED packages have been found:
* ecbuild (required version >= 3)
* Python3
-- The following OPTIONAL packages have not been found:
* oops (required version >= 1.0.0)
* ioda
-- Configuring done
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:269 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:281 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:293 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:304 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:325 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:336 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:347 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:368 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:379 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:411 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:422 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:433 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:444 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:454 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:468 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:483 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:495 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:507 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:648 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:951 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:964 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:975 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:986 (ecbuild_add_test)
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_add_test.cmake:438 (add_test):
Error evaluating generator expression:
$<TARGET_FILE_DIR:_ioda_python>
No target "_ioda_python"
Call Stack (most recent call first):
test/CMakeLists.txt:997 (ecbuild_add_test)
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:
BUILD_IODA_CONVERTERS
CMake Generate step failed. Build files cannot be regenerated correctly.
Hi @LL_GZ,
Unfortunately, we’ve got an issue with cmake export/import between ioda and ioda-converters repositories. (The ioda python API is required by ioda-converters).
Fortunately, we’ve got a workaround which is to build the converters as part of the ioda-bundle. In other words, for now, you need to clone ioda-bundle and use that to build the converters.
git clone https://github.com/jcsda-internal/ioda-bundle
cd ioda-bundle
mkdir build
cd build
ecbuild -DBUILD_IODA_CONVERTERS=ON ..
make -j4
ctest -R ioda
...
Sorry, I can’t find the website https://github.com/jcsda-internal/ioda-bundle. I also searched the ioda-bunde in jcsda-internal repositories, but couldn’t find it either.
@LL_GZ my apologies. You are not in the github jcsda-internal organization and the ioda-bundle repository is private. I think you have two options:
-
If you are going to be working on JEDI in the future, ask to be added to the jcsda-internal organization. To do this you can send the request to Yannick (tremolet@ucar.edu) and include your github user name. I believe that you need to listed as an in-kind in the JCSDA AOP to go this route.
-
If you are not going to be working on JEDI and just need the converter ask someone at NCAR who is in the jcsda-internal organization to build it for you. Check with @snyderdad to see who the best person is for helping you.
Thanks for your patience with this!
@LL_GZ, I got this message from Jamie which might be helpful:
Hi Steve,
My account (hclin) is still on hold by the forum system.
Regarding to the question on the forum about observations, I would
suggest the user try my personal public repository
I know my code is not inline with JEDI rules, but it is easier to use
for general users, I think.
Jamie
@LL_GZ, I’ll second @stephenh’s reply. The stand-alone obs2ioda from Jamie Bresch is the best route if you want to generate additional IODA files.
Thanks for finding this wrinkle in our release. Hopefully ioda-bundle will be made public in an upcoming release.
Unfortunately, I tried obs2ioda1 and obs2ioda2, but both reported errors when running mpasjedi_hofx3d.x.
When I use obs2ioda1, the error reported is:
Radiosonde vars: 4 variables: air_temperature, eastward_wind, northward_wind, specific_humidity
HDF5-DIAG: Error detected in HDF5 (1.12.0) MPI-process 0:
#000: ../../src/H5D.c line 296 in H5Dopen2(): unable to open dataset
major: Dataset
minor: Can't open object
#001: ../../src/H5VLcallback.c line 1974 in H5VL_dataset_open(): dataset open failed
major: Virtual Object Layer
minor: Can't open object
#002: ../../src/H5VLcallback.c line 1941 in H5VL__dataset_open(): dataset open failed
major: Virtual Object Layer
minor: Can't open object
#003: ../../src/H5VLnative_dataset.c line 125 in H5VL__native_dataset_open(): unable to open dataset
major: Dataset
minor: Can't open object
#004: ../../src/H5Dint.c line 1482 in H5D__open_name(): not found
major: Dataset
minor: Object not found
#005: ../../src/H5Gloc.c line 462 in H5G_loc_find(): can't find object
major: Symbol table
minor: Object not found
#006: ../../src/H5Gtraverse.c line 855 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#007: ../../src/H5Gtraverse.c line 744 in H5G__traverse_real(): component not found
major: Symbol table
minor: Object not found
==> destruct MPAS corelist and domain: 0
Exception: Reason: An exception occurred inside ioda while opening a variable.
name: virtual_temperature@ObsError
When I use obs2iodav2 to convert the observation file, the error becomes：
Radiosonde vars: 4 variables: air_temperature, eastward_wind, northward_wind, specific_humidity
WARNING: ObsSpace::createVariables: Skipping variable due to an unexpected data type for variable: MetaData/dateTime
==> destruct MPAS corelist and domain: 0
Exception: Reason: An exception occurred inside ioda while opening a variable.
name: MetaData/dateTime
source_column: 0
source_filename: /fs1/home/tianxj/yhluo/jedi/mpas-bundle/ioda/src/engines/ioda/src/ioda/Has_Variables.cpp
source_function: virtual ioda::Variable ioda::detail::Has_Variables_Base::open(const string&) const
source_line: 107
Exception: oops::HofX3D<MPAS, UFO and IODA observations> terminating...
I compared the nc or h5 files obtained using obs2ioda conversion and they are not quite the same as the observation files given in the tutorial.
@LL_GZ , Would you please put your converted files on google drive? Do you run ioda-upgrade.x after obs2ioda1 ?
run ioda-upgrade.x after obs2ioda1.x:
file converted by obs2ioda2.x:
file converted by obs2ioda1.x:
After I tried ioda-upgrade.x, I encountered the same error as when using only obs2ioda1.
For your tutorial case,
- You cannot use “obs2iodav2” converter for tutorial case. Because it includes epoch time for ”/MetaData/dateTime”, which is not included in tutorial code (it’s the reason you got error when reading “ MetaData/dateTime” from the file sondes_obs_2017081000.h5).
- Your sondes_obs_2017081000.nc4 (from obs2ioda1.x) is not a HDF5 file, so you got failure “unable to open the dataset”.
- Your file “sondes” (from obs2ioda1.x + ioda-upgrade.x) looks correct.
1 Like
This is the correct answer. I finally succeed from obs2ioda1.x and ioda-upgrade.x. Thainks!