I’m seeing a peculiar behavior when running tests to retrieve data while using the develop branch of jedi-bundle. This behavior does not occur when using the 5.0.0 tag/branch.
On Orion, when I run ctest -R get_ for develop I get three failures:
Test project /work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build
Start 505: get_crtm_coeffs
1/7 Test #505: get_crtm_coeffs ........................ Passed 43.06 sec
Start 776: get_ioda_test_data
2/7 Test #776: get_ioda_test_data .....................***Failed 0.13 sec
Start 950: ufo_get_ufo_test_data
3/7 Test #950: ufo_get_ufo_test_data ..................***Failed 0.12 sec
Start 951: ufo_get_crtm_test_data
4/7 Test #951: ufo_get_crtm_test_data ................. Passed 0.25 sec
Start 980: test_ufo_geovals_get_nonexistent_var
5/7 Test #980: test_ufo_geovals_get_nonexistent_var ... Passed 0.19 sec
Start 1422: fv3jedi_get_fv3-jedi_test_data
6/7 Test #1422: fv3jedi_get_fv3-jedi_test_data .........***Failed 0.13 sec
Start 1423: fv3jedi_get_crtm_test_data
7/7 Test #1423: fv3jedi_get_crtm_test_data ............. Passed 0.25 sec
Running the tests with -VV show that they fail because of missing directories. For example:
charrop@Orion-login-4:/work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build> ctest -VV -R get_ioda_test_data
UpdateCTestConfiguration from :/work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build/DartConfiguration.tcl
Parse Config file:/work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build/DartConfiguration.tcl
Parse Config file:/work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build/DartConfiguration.tcl
Test project /work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 776
Start 776: get_ioda_test_data
776: Test command: /work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/build/bin/ioda_data_checker.py "/work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/jedi-bundle/ioda-data"
776: Environment variables:
776: OMP_NUM_THREADS=1
776: Test timeout computed to be: 1500
776: /work/noaa/gsd-hpcs/charrop/SENA/JEDI/develop/jedi-bundle/ioda-data does not exist
1/1 Test #776: get_ioda_test_data ...............***Failed 0.13 sec
0% tests passed, 1 tests failed out of 1
If I create the missing directories for each of those three failed tests, they will all pass. However, the curious thing is that none of the directories gets populated with anything when the tests run. They remain empty. It is unclear why those directories are needed at all.
So… I’m wondering if this behavior is expected or not. It seems to me that the either the tests are broken, or the develop branch is missing some empty directories that should be there. A user should not have to create these. I considered creating an issue in the repository, but as I have been away from JEDI development for a very long time, I thought I’d post it here first. I can open the issue if you want me to.
The complete script:
# Load the stack and environment
module purge
module use /work/noaa/da/role-da/spack-stack/modulefiles
module load miniconda/3.9.7
module load ecflow/5.8.4
module load mysql/8.0.31
#module use /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.4.0/envs/unified-env-v2/install/modulefiles/Core
module use /work/noaa/epic/role-epic/spack-stack/spack-stack-1.4.1/envs/unified-env/install/modulefiles/Core
module load stack-intel/2022.0.2
module load stack-intel-oneapi-mpi/2021.5.1
module load stack-python/3.9.7
module load jedi-fv3-env/unified-dev
module load ewok-env/unified-dev
module load soca-env/unified-dev
module unload crtm
module list
ulimit -s unlimited
ulimit -v unlimited
export SLURM_EXPORT_ENV=ALL
export HDF5_USE_FILE_LOCKING=FALSE
# Create data directories (for some reason develop needs this, but 5.0.0 does not)
# This is a workaround for a bug in tests
mkdir ${JEDI_SRC}/ioda-data
mkdir ${JEDI_SRC}/ufo-data
mkdir ${JEDI_SRC}/fv3-jedi-data
# Build jedi-bundle
cd ${JEDI_BUILD}
ctest -R get_ > ctest.out 2>&1