JEDI Tutorial fv3-bundle ctest bug

JEDI Academy
1

Hi there,

I believe I found a bug while running through the JEDI tutorial where the URL for saber test data in the fv3-bundle is incorrectly specified. This bug likely exists in several packages where they point to dashrepo.ucar.edu in the step to get test data, but this has now been replaced with gdex.ucar.edu. I’ve been able to debug this on my own to get all test cases to pass after building, but I haven’t fully isolated the issue for a true fix. I’m hoping my analysis can help folks who are more familiar with the inter-package structure to apply the proper fixes – my investigation goes as follows in multiple posts due to the limit on characters.

2

In the tutorial, at the point we run the fv3-bundle testsuite, with the command

ctest

from the build directory, I get the following output:

85% tests passed, 159 tests failed out of 1043

Label Time Summary:
HofX            =   1.22 sec*proc (8 tests)
aircraft        =   0.14 sec*proc (1 test)
crtm            =  70.62 sec*proc (81 tests)
executable      =  36.22 sec*proc (145 tests)
femps           =  12.36 sec*proc (1 test)
fv3-jedi        = 785.45 sec*proc (74 tests)
fv3jedi         = 787.70 sec*proc (75 tests)
instrument      =   1.22 sec*proc (8 tests)
ioda            =  30.34 sec*proc (58 tests)
ioda-engines    =  10.67 sec*proc (99 tests)
mpi             = 186.59 sec*proc (201 tests)
oops            =  62.55 sec*proc (221 tests)
openmp          =  96.02 sec*proc (132 tests)
radarVAD        =   0.07 sec*proc (1 test)
rass            =   0.28 sec*proc (1 test)
saber           =  81.90 sec*proc (164 tests)
satwinds        =   0.06 sec*proc (1 test)
scatwinds       =   0.06 sec*proc (1 test)
script          = 1540.36 sec*proc (801 tests)
sfcLand         =   0.08 sec*proc (1 test)
sfcMarine       =   0.30 sec*proc (1 test)
sonde           =   0.23 sec*proc (1 test)
ufo             = 531.20 sec*proc (347 tests)

Total Test time (real) = 1589.53 sec

The following tests FAILED:
	305 - get_data_saber_data.tar.gz (Failed)
	306 - get_data_saber_data_mpi.tar.gz (Failed)
	307 - get_data_saber_data_oops.tar.gz (Failed)
	308 - get_data_saber_ref_1.tar.gz (Failed)
	309 - get_data_saber_ref_mpi_1.tar.gz (Failed)
	310 - get_data_saber_ref_cgal.tar.gz (Failed)
	311 - get_data_saber_ref_mpi_cgal.tar.gz (Failed)
	312 - get_data_saber_ref_oops.tar.gz (Failed)
	313 - test_bump_normality_1-1_run (Failed)
	314 - test_bump_normality_1-1_compare (Failed)
	315 - test_bump_hdiag_cor_diag_all_1-1_run (Failed)
	316 - test_bump_hdiag_cor_diag_all_1-1_compare (Failed)
	317 - test_bump_hdiag_grid_1-1_run (Failed)
	318 - test_bump_hdiag_grid_1-1_compare (Failed)
	319 - test_bump_hdiag_hyb-rnd_common_1-1_run (Failed)
	320 - test_bump_hdiag_hyb-rnd_common_1-1_compare (Failed)
	321 - test_bump_hdiag_mask_stddev_upper_1-1_run (Failed)
	322 - test_bump_hdiag_mask_stddev_upper_1-1_compare (Failed)
	323 - test_bump_hdiag-nicas_cor_specific_univariate_1-1_run (Failed)
	324 - test_bump_hdiag-nicas_cor_specific_univariate_1-1_compare (Failed)
	325 - test_bump_hdiag-nicas_local_diag_1-1_run (Failed)
	326 - test_bump_hdiag-nicas_local_diag_1-1_compare (Failed)
	327 - test_bump_hdiag-nicas_loc_common_1-1_run (Failed)
	328 - test_bump_hdiag-nicas_loc_common_1-1_compare (Failed)
	329 - test_bump_hdiag-nicas_mask_check_1-1_run (Failed)
	330 - test_bump_hdiag-nicas_mask_check_1-1_compare (Failed)
	331 - test_bump_hdiag-nicas_mesh_based_1-1_run (Failed)
	332 - test_bump_hdiag-nicas_mesh_based_1-1_compare (Failed)
	333 - test_bump_hdiag-nicas_network_1-1_run (Failed)
	334 - test_bump_hdiag-nicas_network_1-1_compare (Failed)
	335 - test_bump_lct-nicas_one_scale_1-1_run (Failed)
	336 - test_bump_lct-nicas_one_scale_1-1_compare (Failed)
	337 - test_bump_nicas_fast_sampling_1-1_run (Failed)
	338 - test_bump_nicas_fast_sampling_1-1_compare (Failed)
	339 - test_bump_nicas_subsamp_hvh_1-1_run (Failed)
	340 - test_bump_nicas_subsamp_hvh_1-1_compare (Failed)
	341 - test_bump_read_cmat_1-1_run (Failed)
	342 - test_bump_read_cmat_1-1_compare (Failed)
	343 - test_bump_read_cmat_serial_1-1_run (Failed)
	344 - test_bump_read_cmat_serial_1-1_compare (Failed)
	345 - test_bump_read_mom_1-1_run (Failed)
	346 - test_bump_read_mom_1-1_compare (Failed)
	347 - test_bump_read_nicas_1-1_run (Failed)
	348 - test_bump_read_nicas_1-1_compare (Failed)
	349 - test_bump_read_sampling_1-1_run (Failed)
	350 - test_bump_read_sampling_1-1_compare (Failed)
	351 - test_bump_read_var_1-1_run (Failed)
	352 - test_bump_read_var_1-1_compare (Failed)
	353 - test_bump_read_vbal_1-1_run (Failed)
	354 - test_bump_read_vbal_1-1_compare (Failed)
	355 - test_bump_var_forced_var_1-1_run (Failed)
	356 - test_bump_var_forced_var_1-1_compare (Failed)
	357 - test_bump_vbal_1-1_run (Failed)
	358 - test_bump_vbal_1-1_compare (Failed)
	359 - test_bump_write_cmat_1-1_run (Failed)
	360 - test_bump_write_cmat_1-1_compare (Failed)
	361 - test_bump_write_cmat_serial_1-1_run (Failed)
	362 - test_bump_write_cmat_serial_1-1_compare (Failed)
	363 - test_bump_write_mom_1-1_run (Failed)
	364 - test_bump_write_mom_1-1_compare (Failed)
	365 - test_bump_write_nicas_1-1_run (Failed)
	366 - test_bump_write_nicas_1-1_compare (Failed)
	367 - test_bump_write_sampling_1-1_run (Failed)
	368 - test_bump_write_sampling_1-1_compare (Failed)
	369 - test_bump_write_var_1-1_run (Failed)
	370 - test_bump_write_var_1-1_compare (Failed)
	371 - test_bump_write_vbal_1-1_run (Failed)
	372 - test_bump_write_vbal_1-1_compare (Failed)
	373 - test_bump_hdiag-nicas_use_cgal_1-1_run (Failed)
	374 - test_bump_hdiag-nicas_use_cgal_1-1_compare (Failed)
	375 - test_bump_normality_2-1_run (Failed)
	376 - test_bump_normality_2-1_compare (Failed)
	377 - test_bump_hdiag_cor_diag_all_2-1_run (Failed)
	378 - test_bump_hdiag_cor_diag_all_2-1_compare (Failed)
	379 - test_bump_hdiag_grid_2-1_run (Failed)
	380 - test_bump_hdiag_grid_2-1_compare (Failed)
	381 - test_bump_hdiag_hyb-rnd_common_2-1_run (Failed)
	382 - test_bump_hdiag_hyb-rnd_common_2-1_compare (Failed)
	383 - test_bump_hdiag_mask_stddev_upper_2-1_run (Failed)
	384 - test_bump_hdiag_mask_stddev_upper_2-1_compare (Failed)
	385 - test_bump_hdiag-nicas_cor_specific_univariate_2-1_run (Failed)
	386 - test_bump_hdiag-nicas_cor_specific_univariate_2-1_compare (Failed)
	387 - test_bump_hdiag-nicas_local_diag_2-1_run (Failed)
	388 - test_bump_hdiag-nicas_local_diag_2-1_compare (Failed)
	389 - test_bump_hdiag-nicas_loc_common_2-1_run (Failed)
	390 - test_bump_hdiag-nicas_loc_common_2-1_compare (Failed)
	391 - test_bump_hdiag-nicas_mask_check_2-1_run (Failed)
	392 - test_bump_hdiag-nicas_mask_check_2-1_compare (Failed)
	393 - test_bump_hdiag-nicas_mesh_based_2-1_run (Failed)
	394 - test_bump_hdiag-nicas_mesh_based_2-1_compare (Failed)
	395 - test_bump_hdiag-nicas_network_2-1_run (Failed)
	396 - test_bump_hdiag-nicas_network_2-1_compare (Failed)
	397 - test_bump_lct-nicas_one_scale_2-1_run (Failed)
	398 - test_bump_lct-nicas_one_scale_2-1_compare (Failed)
	399 - test_bump_nicas_fast_sampling_2-1_run (Failed)
	400 - test_bump_nicas_fast_sampling_2-1_compare (Failed)
	401 - test_bump_nicas_subsamp_hvh_2-1_run (Failed)
	402 - test_bump_nicas_subsamp_hvh_2-1_compare (Failed)
	403 - test_bump_read_cmat_2-1_run (Failed)
	404 - test_bump_read_cmat_2-1_compare (Failed)
	405 - test_bump_read_cmat_serial_2-1_run (Failed)
	406 - test_bump_read_cmat_serial_2-1_compare (Failed)
	407 - test_bump_read_mom_2-1_run (Failed)
	408 - test_bump_read_mom_2-1_compare (Failed)
	409 - test_bump_read_nicas_2-1_run (Failed)
	410 - test_bump_read_nicas_2-1_compare (Failed)
	411 - test_bump_read_sampling_2-1_run (Failed)
	412 - test_bump_read_sampling_2-1_compare (Failed)
	413 - test_bump_read_var_2-1_run (Failed)
	414 - test_bump_read_var_2-1_compare (Failed)
	415 - test_bump_read_vbal_2-1_run (Failed)
	416 - test_bump_read_vbal_2-1_compare (Failed)
	417 - test_bump_var_forced_var_2-1_run (Failed)
	418 - test_bump_var_forced_var_2-1_compare (Failed)
	419 - test_bump_vbal_2-1_run (Failed)
	420 - test_bump_vbal_2-1_compare (Failed)
	421 - test_bump_write_cmat_2-1_run (Failed)
	422 - test_bump_write_cmat_2-1_compare (Failed)
	423 - test_bump_write_cmat_serial_2-1_run (Failed)
	424 - test_bump_write_cmat_serial_2-1_compare (Failed)
	425 - test_bump_write_mom_2-1_run (Failed)
	426 - test_bump_write_mom_2-1_compare (Failed)
	427 - test_bump_write_nicas_2-1_run (Failed)
	428 - test_bump_write_nicas_2-1_compare (Failed)
	429 - test_bump_write_sampling_2-1_run (Failed)
	430 - test_bump_write_sampling_2-1_compare (Failed)
	431 - test_bump_write_var_2-1_run (Failed)
	432 - test_bump_write_var_2-1_compare (Failed)
	433 - test_bump_write_vbal_2-1_run (Failed)
	434 - test_bump_write_vbal_2-1_compare (Failed)
	435 - test_bump_hdiag-nicas_use_cgal_2-1_run (Failed)
	436 - test_bump_hdiag-nicas_use_cgal_2-1_compare (Failed)
	437 - test_bump_read_cmat_parallel-serial_compare (Failed)
	438 - test_bump_write_cmat_parallel-serial_compare (Failed)
	439 - test_qg_ens_recenter (Failed)
	440 - test_qg_parameters_bump_cov (Failed)
	441 - test_qg_parameters_bump_cov_pairs (Failed)
	442 - test_qg_parameters_bump_cov_incens (Failed)
	443 - test_qg_parameters_bump_cov_no_ens (Failed)
	444 - test_qg_parameters_bump_cov_nsub (Failed)
	445 - test_qg_parameters_bump_hyb (Failed)
	446 - test_qg_parameters_bump_lct (Failed)
	447 - test_qg_parameters_bump_loc (Failed)
	449 - test_qg_3densvar_bump (Failed)
	450 - test_qg_3dvar_bump (Failed)
	451 - test_qg_3dvar_hybrid_bump (Failed)
	452 - test_qg_4densvar_bump (Failed)
	453 - test_qg_4dvar_drplanczos_bump (Failed)
	454 - test_qg_4dvar_drplanczos_hybrid_bump (Failed)
	455 - test_qg_dirac_bump_cov (Failed)
	456 - test_qg_dirac_bump_cov_no_ens (Failed)
	457 - test_qg_dirac_bump_hyb_field (Failed)
	458 - test_qg_dirac_bump_hyb_value (Failed)
	459 - test_qg_dirac_bump_lct (Failed)
	460 - test_qg_dirac_bump_loc_3d (Failed)
	461 - test_qg_dirac_bump_loc_4d (Failed)
	462 - test_qg_dirac_id_cov (Failed)
	463 - test_qg_dirac_id_loc_3d (Failed)
	464 - test_qg_dirac_id_loc_4d (Failed)
3

Re-running verbosely as

ctest --rerun-failed --output-on-failure

I noticed that we have the following error in the first failed case:

        Start 305: get_data_saber_data.tar.gz
  1/159 Test #305: get_data_saber_data.tar.gz ..................................***Failed    0.41 sec
downloading RELEASE data from https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_data.tar.gz
Traceback (most recent call last):
  File "/worktmp/modified_build/bin/saber_data_downloader.py", line 55, in <module>
    DownloadUntar(download_base_url, testfiles_path, testfiles_name, md5check)
  File "/worktmp/modified_build/bin/saber_data_downloader.py", line 20, in DownloadUntar
    urllib.request.urlretrieve( download_base_url+"/"+testfiles_name, testfiles_path+"/"+testfiles_name)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500:

which points to a server-side issue.

4

Entering the URL for the downloader, I noticed that the website states:

The DASH Repository has been replaced by the new Geoscience Data Exchange (GDEX).
Please contact gdex@ucar.edu if you have any questions.

5

This led me to try the following – I make a fresh clone and a modified pre-build as

git clone https://github.com/jcsda/fv3-bundle.git
mkdir modified_build
cd modified_build
ecbuild --build=Release ../fv3-bundle
6

Now I want to verify which of the pulled packages has an incorrectly specified url:

cd ../fv3-bundle
grep -r dashrepo.ucar.edu *

with the output:

crtm/test/CMakeLists.txt:  set( ECBUILD_DOWNLOAD_BASE_URL https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file )
fv3-jedi/test/CMakeLists.txt:    set( ECBUILD_DOWNLOAD_BASE_URL https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file )
ioda/test/CMakeLists.txt:  set( IODA_DOWNLOAD_BASE_URL https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file )
saber/test/CMakeLists.txt:        set( ECBUILD_DOWNLOAD_BASE_URL https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file )
tutorials/Hofx/get_input.bash:BASE_URL=https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file
ufo/test/CMakeLists.txt:    set( ECBUILD_DOWNLOAD_BASE_URL https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file )
7

I replace the dashrepo reference as follows:

find ./ -type f -exec sed -i 's/dashrepo.ucar.edu/gdex.ucar.edu/g' {} \;

Going back to the modified pre-build I grep

cd ../modified_build/
grep -r dashrepo.ucar.edu *

with the output

crtm/test/CTestTestfile.cmake:add_test(get_crtm_coeffs "/worktmp/modified_build/bin/crtm_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/modified_build/test_data/crtm/2.3.0" "crtm_coefficients.tar.gz" "0")
fv3-jedi/test/CTestTestfile.cmake:add_test(fv3jedi_get_fv3-jedi_test_data "/worktmp/modified_build/bin/fv3jedi_testdata_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release" "fv3-jedi_testinput_tier_1.tar.gz" "0")
fv3-jedi/test/CTestTestfile.cmake:add_test(fv3jedi_get_crtm_test_data "/worktmp/modified_build/bin/fv3jedi_testdata_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release/crtm/2.3.0" "crtm_coefficients.tar.gz" "0")
ioda/test/CTestTestfile.cmake:add_test(get_ioda_test_data "/worktmp/modified_build/bin/ioda_dash_downloader.py.in" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release" "ioda_testinput_tier_1.tar.gz" "2.0.2" "ioda")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_data.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_data_mpi.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data_mpi.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_data_oops.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data_oops.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_1.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_1.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_mpi_1.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_mpi_1.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_cgal.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_cgal.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_mpi_cgal.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_mpi_cgal.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_oops.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_oops.tar.gz" "0")
ufo/test/CTestTestfile.cmake:add_test(ufo_get_ufo_test_data "/worktmp/modified_build/bin/ufo_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release" "ufo_testinput_tier_1.tar.gz" "0")
ufo/test/CTestTestfile.cmake:add_test(ufo_get_crtm_test_data "/worktmp/modified_build/bin/ufo_data_downloader.py" "https://dashrepo.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release/crtm/2.3.0" "crtm_coefficients.tar.gz" "0")

and I replace the references again with:

find ./ -type f -exec sed -i 's/dashrepo.ucar.edu/gdex.ucar.edu/g' {} \;
8

Now, running the build / test:

make -j4
ctest

I still get the output:

85% tests passed, 159 tests failed out of 1043

Label Time Summary:
HofX            =   1.22 sec*proc (8 tests)
aircraft        =   0.14 sec*proc (1 test)
crtm            =  70.62 sec*proc (81 tests)
executable      =  36.22 sec*proc (145 tests)
femps           =  12.36 sec*proc (1 test)
fv3-jedi        = 785.45 sec*proc (74 tests)
fv3jedi         = 787.70 sec*proc (75 tests)
instrument      =   1.22 sec*proc (8 tests)
ioda            =  30.34 sec*proc (58 tests)
ioda-engines    =  10.67 sec*proc (99 tests)
mpi             = 186.59 sec*proc (201 tests)
oops            =  62.55 sec*proc (221 tests)
openmp          =  96.02 sec*proc (132 tests)
radarVAD        =   0.07 sec*proc (1 test)
rass            =   0.28 sec*proc (1 test)
saber           =  81.90 sec*proc (164 tests)
satwinds        =   0.06 sec*proc (1 test)
scatwinds       =   0.06 sec*proc (1 test)
script          = 1540.36 sec*proc (801 tests)
sfcLand         =   0.08 sec*proc (1 test)
sfcMarine       =   0.30 sec*proc (1 test)
sonde           =   0.23 sec*proc (1 test)
ufo             = 531.20 sec*proc (347 tests)

Total Test time (real) = 1589.53 sec

The following tests FAILED:
	305 - get_data_saber_data.tar.gz (Failed)
	306 - get_data_saber_data_mpi.tar.gz (Failed)
	307 - get_data_saber_data_oops.tar.gz (Failed)
	308 - get_data_saber_ref_1.tar.gz (Failed)
	309 - get_data_saber_ref_mpi_1.tar.gz (Failed)
	310 - get_data_saber_ref_cgal.tar.gz (Failed)
	311 - get_data_saber_ref_mpi_cgal.tar.gz (Failed)
	312 - get_data_saber_ref_oops.tar.gz (Failed)
	313 - test_bump_normality_1-1_run (Failed)
	314 - test_bump_normality_1-1_compare (Failed)
	315 - test_bump_hdiag_cor_diag_all_1-1_run (Failed)
	316 - test_bump_hdiag_cor_diag_all_1-1_compare (Failed)
	317 - test_bump_hdiag_grid_1-1_run (Failed)
	318 - test_bump_hdiag_grid_1-1_compare (Failed)
	319 - test_bump_hdiag_hyb-rnd_common_1-1_run (Failed)
	320 - test_bump_hdiag_hyb-rnd_common_1-1_compare (Failed)
	321 - test_bump_hdiag_mask_stddev_upper_1-1_run (Failed)
	322 - test_bump_hdiag_mask_stddev_upper_1-1_compare (Failed)
	323 - test_bump_hdiag-nicas_cor_specific_univariate_1-1_run (Failed)
	324 - test_bump_hdiag-nicas_cor_specific_univariate_1-1_compare (Failed)
	325 - test_bump_hdiag-nicas_local_diag_1-1_run (Failed)
	326 - test_bump_hdiag-nicas_local_diag_1-1_compare (Failed)
	327 - test_bump_hdiag-nicas_loc_common_1-1_run (Failed)
	328 - test_bump_hdiag-nicas_loc_common_1-1_compare (Failed)
	329 - test_bump_hdiag-nicas_mask_check_1-1_run (Failed)
	330 - test_bump_hdiag-nicas_mask_check_1-1_compare (Failed)
	331 - test_bump_hdiag-nicas_mesh_based_1-1_run (Failed)
	332 - test_bump_hdiag-nicas_mesh_based_1-1_compare (Failed)
	333 - test_bump_hdiag-nicas_network_1-1_run (Failed)
	334 - test_bump_hdiag-nicas_network_1-1_compare (Failed)
	335 - test_bump_lct-nicas_one_scale_1-1_run (Failed)
	336 - test_bump_lct-nicas_one_scale_1-1_compare (Failed)
	337 - test_bump_nicas_fast_sampling_1-1_run (Failed)
	338 - test_bump_nicas_fast_sampling_1-1_compare (Failed)
	339 - test_bump_nicas_subsamp_hvh_1-1_run (Failed)
	340 - test_bump_nicas_subsamp_hvh_1-1_compare (Failed)
	341 - test_bump_read_cmat_1-1_run (Failed)
	342 - test_bump_read_cmat_1-1_compare (Failed)
	343 - test_bump_read_cmat_serial_1-1_run (Failed)
	344 - test_bump_read_cmat_serial_1-1_compare (Failed)
	345 - test_bump_read_mom_1-1_run (Failed)
	346 - test_bump_read_mom_1-1_compare (Failed)
	347 - test_bump_read_nicas_1-1_run (Failed)
	348 - test_bump_read_nicas_1-1_compare (Failed)
	349 - test_bump_read_sampling_1-1_run (Failed)
	350 - test_bump_read_sampling_1-1_compare (Failed)
	351 - test_bump_read_var_1-1_run (Failed)
	352 - test_bump_read_var_1-1_compare (Failed)
	353 - test_bump_read_vbal_1-1_run (Failed)
	354 - test_bump_read_vbal_1-1_compare (Failed)
	355 - test_bump_var_forced_var_1-1_run (Failed)
	356 - test_bump_var_forced_var_1-1_compare (Failed)
	357 - test_bump_vbal_1-1_run (Failed)
	358 - test_bump_vbal_1-1_compare (Failed)
	359 - test_bump_write_cmat_1-1_run (Failed)
	360 - test_bump_write_cmat_1-1_compare (Failed)
	361 - test_bump_write_cmat_serial_1-1_run (Failed)
	362 - test_bump_write_cmat_serial_1-1_compare (Failed)
	363 - test_bump_write_mom_1-1_run (Failed)
	364 - test_bump_write_mom_1-1_compare (Failed)
	365 - test_bump_write_nicas_1-1_run (Failed)
	366 - test_bump_write_nicas_1-1_compare (Failed)
	367 - test_bump_write_sampling_1-1_run (Failed)
	368 - test_bump_write_sampling_1-1_compare (Failed)
	369 - test_bump_write_var_1-1_run (Failed)
	370 - test_bump_write_var_1-1_compare (Failed)
	371 - test_bump_write_vbal_1-1_run (Failed)
	372 - test_bump_write_vbal_1-1_compare (Failed)
	373 - test_bump_hdiag-nicas_use_cgal_1-1_run (Failed)
	374 - test_bump_hdiag-nicas_use_cgal_1-1_compare (Failed)
	375 - test_bump_normality_2-1_run (Failed)
	376 - test_bump_normality_2-1_compare (Failed)
	377 - test_bump_hdiag_cor_diag_all_2-1_run (Failed)
	378 - test_bump_hdiag_cor_diag_all_2-1_compare (Failed)
	379 - test_bump_hdiag_grid_2-1_run (Failed)
	380 - test_bump_hdiag_grid_2-1_compare (Failed)
	381 - test_bump_hdiag_hyb-rnd_common_2-1_run (Failed)
	382 - test_bump_hdiag_hyb-rnd_common_2-1_compare (Failed)
	383 - test_bump_hdiag_mask_stddev_upper_2-1_run (Failed)
	384 - test_bump_hdiag_mask_stddev_upper_2-1_compare (Failed)
	385 - test_bump_hdiag-nicas_cor_specific_univariate_2-1_run (Failed)
	386 - test_bump_hdiag-nicas_cor_specific_univariate_2-1_compare (Failed)
	387 - test_bump_hdiag-nicas_local_diag_2-1_run (Failed)
	388 - test_bump_hdiag-nicas_local_diag_2-1_compare (Failed)
	389 - test_bump_hdiag-nicas_loc_common_2-1_run (Failed)
	390 - test_bump_hdiag-nicas_loc_common_2-1_compare (Failed)
	391 - test_bump_hdiag-nicas_mask_check_2-1_run (Failed)
	392 - test_bump_hdiag-nicas_mask_check_2-1_compare (Failed)
	393 - test_bump_hdiag-nicas_mesh_based_2-1_run (Failed)
	394 - test_bump_hdiag-nicas_mesh_based_2-1_compare (Failed)
	395 - test_bump_hdiag-nicas_network_2-1_run (Failed)
	396 - test_bump_hdiag-nicas_network_2-1_compare (Failed)
	397 - test_bump_lct-nicas_one_scale_2-1_run (Failed)
	398 - test_bump_lct-nicas_one_scale_2-1_compare (Failed)
	399 - test_bump_nicas_fast_sampling_2-1_run (Failed)
	400 - test_bump_nicas_fast_sampling_2-1_compare (Failed)
	401 - test_bump_nicas_subsamp_hvh_2-1_run (Failed)
	402 - test_bump_nicas_subsamp_hvh_2-1_compare (Failed)
	403 - test_bump_read_cmat_2-1_run (Failed)
	404 - test_bump_read_cmat_2-1_compare (Failed)
	405 - test_bump_read_cmat_serial_2-1_run (Failed)
	406 - test_bump_read_cmat_serial_2-1_compare (Failed)
	407 - test_bump_read_mom_2-1_run (Failed)
	408 - test_bump_read_mom_2-1_compare (Failed)
	409 - test_bump_read_nicas_2-1_run (Failed)
	410 - test_bump_read_nicas_2-1_compare (Failed)
	411 - test_bump_read_sampling_2-1_run (Failed)
	412 - test_bump_read_sampling_2-1_compare (Failed)
	413 - test_bump_read_var_2-1_run (Failed)
	414 - test_bump_read_var_2-1_compare (Failed)
	415 - test_bump_read_vbal_2-1_run (Failed)
	416 - test_bump_read_vbal_2-1_compare (Failed)
	417 - test_bump_var_forced_var_2-1_run (Failed)
	418 - test_bump_var_forced_var_2-1_compare (Failed)
	419 - test_bump_vbal_2-1_run (Failed)
	420 - test_bump_vbal_2-1_compare (Failed)
	421 - test_bump_write_cmat_2-1_run (Failed)
	422 - test_bump_write_cmat_2-1_compare (Failed)
	423 - test_bump_write_cmat_serial_2-1_run (Failed)
	424 - test_bump_write_cmat_serial_2-1_compare (Failed)
	425 - test_bump_write_mom_2-1_run (Failed)
	426 - test_bump_write_mom_2-1_compare (Failed)
	427 - test_bump_write_nicas_2-1_run (Failed)
	428 - test_bump_write_nicas_2-1_compare (Failed)
	429 - test_bump_write_sampling_2-1_run (Failed)
	430 - test_bump_write_sampling_2-1_compare (Failed)
	431 - test_bump_write_var_2-1_run (Failed)
	432 - test_bump_write_var_2-1_compare (Failed)
	433 - test_bump_write_vbal_2-1_run (Failed)
	434 - test_bump_write_vbal_2-1_compare (Failed)
	435 - test_bump_hdiag-nicas_use_cgal_2-1_run (Failed)
	436 - test_bump_hdiag-nicas_use_cgal_2-1_compare (Failed)
	437 - test_bump_read_cmat_parallel-serial_compare (Failed)
	438 - test_bump_write_cmat_parallel-serial_compare (Failed)
	439 - test_qg_ens_recenter (Failed)
	440 - test_qg_parameters_bump_cov (Failed)
	441 - test_qg_parameters_bump_cov_pairs (Failed)
	442 - test_qg_parameters_bump_cov_incens (Failed)
	443 - test_qg_parameters_bump_cov_no_ens (Failed)
	444 - test_qg_parameters_bump_cov_nsub (Failed)
	445 - test_qg_parameters_bump_hyb (Failed)
	446 - test_qg_parameters_bump_lct (Failed)
	447 - test_qg_parameters_bump_loc (Failed)
	449 - test_qg_3densvar_bump (Failed)
	450 - test_qg_3dvar_bump (Failed)
	451 - test_qg_3dvar_hybrid_bump (Failed)
	452 - test_qg_4densvar_bump (Failed)
	453 - test_qg_4dvar_drplanczos_bump (Failed)
	454 - test_qg_4dvar_drplanczos_hybrid_bump (Failed)
	455 - test_qg_dirac_bump_cov (Failed)
	456 - test_qg_dirac_bump_cov_no_ens (Failed)
	457 - test_qg_dirac_bump_hyb_field (Failed)
	458 - test_qg_dirac_bump_hyb_value (Failed)
	459 - test_qg_dirac_bump_lct (Failed)
	460 - test_qg_dirac_bump_loc_3d (Failed)
	461 - test_qg_dirac_bump_loc_4d (Failed)
	462 - test_qg_dirac_id_cov (Failed)
	463 - test_qg_dirac_id_loc_3d (Failed)
	464 - test_qg_dirac_id_loc_4d (Failed)
9

Re-running verbosely, we see the first failure with the message:

        Start 305: get_data_saber_data.tar.gz
  1/159 Test #305: get_data_saber_data.tar.gz ..................................***Failed    0.41 sec
downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_data.tar.gz
Traceback (most recent call last):
  File "/worktmp/modified_build/bin/saber_data_downloader.py", line 55, in <module>
    DownloadUntar(download_base_url, testfiles_path, testfiles_name, md5check)
  File "/worktmp/modified_build/bin/saber_data_downloader.py", line 20, in DownloadUntar
    urllib.request.urlretrieve( download_base_url+"/"+testfiles_name, testfiles_path+"/"+testfiles_name)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/miniconda3/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500:

Typing in the URL again, I noticed that the two slashes put together in

/file//saber_data.tar.gz

is still causing an error, where the URL:

https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/saber_data.tar.gz

does point to the proper data.

I spent some time looking for where the name reference is passed to the base URL, and the best I could come up with is that it seems like

testfiles_name = sys.argv[3]

is specified incorrectly (pre-pended with a slash) when it is used in the function call of

DownloadUntar

from the

fv3-bundle/saber/test/saber_data_downloader.py

module. Without knowing where the

sys.argv[3]

is actually specified in the build / test, I decided to just see if I could make a manual fix. In the modified build, I call

grep -r gdex.ucar.edu *

with the output

crtm/test/CTestTestfile.cmake:add_test(get_crtm_coeffs "/worktmp/modified_build/bin/crtm_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/modified_build/test_data/crtm/2.3.0" "crtm_coefficients.tar.gz" "0")
fv3-jedi/test/CTestTestfile.cmake:add_test(fv3jedi_get_fv3-jedi_test_data "/worktmp/modified_build/bin/fv3jedi_testdata_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release" "fv3-jedi_testinput_tier_1.tar.gz" "0")
fv3-jedi/test/CTestTestfile.cmake:add_test(fv3jedi_get_crtm_test_data "/worktmp/modified_build/bin/fv3jedi_testdata_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release/crtm/2.3.0" "crtm_coefficients.tar.gz" "0")
ioda/test/CTestTestfile.cmake:add_test(get_ioda_test_data "/worktmp/modified_build/bin/ioda_dash_downloader.py.in" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release" "ioda_testinput_tier_1.tar.gz" "2.0.2" "ioda")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_data.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_data_mpi.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data_mpi.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_data_oops.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data_oops.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_1.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_1.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_mpi_1.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_mpi_1.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_cgal.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_cgal.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_mpi_cgal.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_mpi_cgal.tar.gz" "0")
saber/test/CTestTestfile.cmake:add_test(get_data_saber_ref_oops.tar.gz "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_oops.tar.gz" "0")
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_data.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data_mpi.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_data_mpi.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testdata" "saber_data_oops.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_data_oops.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_1.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_ref_1.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_mpi_1.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_ref_mpi_1.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_cgal.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_ref_cgal.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_mpi_cgal.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_ref_mpi_cgal.tar.gz
Testing/Temporary/LastTest.log:Command: "/worktmp/modified_build/bin/saber_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/" "/worktmp/modified_build/test_data/saber/1.1.1/testref" "saber_ref_oops.tar.gz" "0"
Testing/Temporary/LastTest.log:downloading RELEASE data from https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file//saber_ref_oops.tar.gz
ufo/test/CTestTestfile.cmake:add_test(ufo_get_ufo_test_data "/worktmp/modified_build/bin/ufo_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release" "ufo_testinput_tier_1.tar.gz" "0")
ufo/test/CTestTestfile.cmake:add_test(ufo_get_crtm_test_data "/worktmp/modified_build/bin/ufo_data_downloader.py" "https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file" "/worktmp/fv3-bundle/test-data-release/crtm/2.3.0" "crtm_coefficients.tar.gz" "0")

I notice that all of the erroneous double slash URLs are in the file

saber/test/CTestTestfile.cmake

Manually removing the trailing slash from the strings:

"https://gdex.ucar.edu/api/v1/dataset/147_miesch/version/1.1.0/file/"

and re-running the tests

ctest --rerun-failed --output-on-failure

I get the following output:

Test project /worktmp/modified_build
        Start 305: get_data_saber_data.tar.gz
  1/159 Test #305: get_data_saber_data.tar.gz ..................................   Passed    2.20 sec
        Start 306: get_data_saber_data_mpi.tar.gz
  2/159 Test #306: get_data_saber_data_mpi.tar.gz ..............................   Passed    1.00 sec
        Start 307: get_data_saber_data_oops.tar.gz
  3/159 Test #307: get_data_saber_data_oops.tar.gz .............................   Passed    4.22 sec
        Start 308: get_data_saber_ref_1.tar.gz
  4/159 Test #308: get_data_saber_ref_1.tar.gz .................................   Passed    2.69 sec
        Start 309: get_data_saber_ref_mpi_1.tar.gz
  5/159 Test #309: get_data_saber_ref_mpi_1.tar.gz .............................   Passed    1.98 sec
        Start 310: get_data_saber_ref_cgal.tar.gz
  6/159 Test #310: get_data_saber_ref_cgal.tar.gz ..............................   Passed    0.73 sec
        Start 311: get_data_saber_ref_mpi_cgal.tar.gz
  7/159 Test #311: get_data_saber_ref_mpi_cgal.tar.gz ..........................   Passed    0.63 sec
        Start 312: get_data_saber_ref_oops.tar.gz
  8/159 Test #312: get_data_saber_ref_oops.tar.gz ..............................   Passed    0.75 sec
        Start 313: test_bump_normality_1-1_run
  9/159 Test #313: test_bump_normality_1-1_run .................................   Passed    5.88 sec
        Start 314: test_bump_normality_1-1_compare
 10/159 Test #314: test_bump_normality_1-1_compare .............................   Passed    0.14 sec
        Start 315: test_bump_hdiag_cor_diag_all_1-1_run
 11/159 Test #315: test_bump_hdiag_cor_diag_all_1-1_run ........................   Passed    1.50 sec
        Start 316: test_bump_hdiag_cor_diag_all_1-1_compare
 12/159 Test #316: test_bump_hdiag_cor_diag_all_1-1_compare ....................   Passed    0.11 sec
        Start 317: test_bump_hdiag_grid_1-1_run
 13/159 Test #317: test_bump_hdiag_grid_1-1_run ................................   Passed    4.07 sec
        Start 318: test_bump_hdiag_grid_1-1_compare
 14/159 Test #318: test_bump_hdiag_grid_1-1_compare ............................   Passed    0.16 sec
        Start 319: test_bump_hdiag_hyb-rnd_common_1-1_run
 15/159 Test #319: test_bump_hdiag_hyb-rnd_common_1-1_run ......................   Passed    1.65 sec
        Start 320: test_bump_hdiag_hyb-rnd_common_1-1_compare
 16/159 Test #320: test_bump_hdiag_hyb-rnd_common_1-1_compare ..................   Passed    0.18 sec
        Start 321: test_bump_hdiag_mask_stddev_upper_1-1_run
 17/159 Test #321: test_bump_hdiag_mask_stddev_upper_1-1_run ...................   Passed    1.37 sec
        Start 322: test_bump_hdiag_mask_stddev_upper_1-1_compare
 18/159 Test #322: test_bump_hdiag_mask_stddev_upper_1-1_compare ...............   Passed    0.09 sec
        Start 323: test_bump_hdiag-nicas_cor_specific_univariate_1-1_run
 19/159 Test #323: test_bump_hdiag-nicas_cor_specific_univariate_1-1_run .......   Passed    5.04 sec
        Start 324: test_bump_hdiag-nicas_cor_specific_univariate_1-1_compare
 20/159 Test #324: test_bump_hdiag-nicas_cor_specific_univariate_1-1_compare ...   Passed    0.38 sec
        Start 325: test_bump_hdiag-nicas_local_diag_1-1_run
 21/159 Test #325: test_bump_hdiag-nicas_local_diag_1-1_run ....................   Passed   12.22 sec
        Start 326: test_bump_hdiag-nicas_local_diag_1-1_compare
 22/159 Test #326: test_bump_hdiag-nicas_local_diag_1-1_compare ................   Passed    0.44 sec
        Start 327: test_bump_hdiag-nicas_loc_common_1-1_run
 23/159 Test #327: test_bump_hdiag-nicas_loc_common_1-1_run ....................   Passed    5.55 sec
        Start 328: test_bump_hdiag-nicas_loc_common_1-1_compare
 24/159 Test #328: test_bump_hdiag-nicas_loc_common_1-1_compare ................   Passed    0.34 sec
        Start 329: test_bump_hdiag-nicas_mask_check_1-1_run
 25/159 Test #329: test_bump_hdiag-nicas_mask_check_1-1_run ....................   Passed    4.50 sec
        Start 330: test_bump_hdiag-nicas_mask_check_1-1_compare
 26/159 Test #330: test_bump_hdiag-nicas_mask_check_1-1_compare ................   Passed    0.28 sec
        Start 331: test_bump_hdiag-nicas_mesh_based_1-1_run
 27/159 Test #331: test_bump_hdiag-nicas_mesh_based_1-1_run ....................   Passed    6.61 sec
        Start 332: test_bump_hdiag-nicas_mesh_based_1-1_compare
 28/159 Test #332: test_bump_hdiag-nicas_mesh_based_1-1_compare ................   Passed    0.39 sec
        Start 333: test_bump_hdiag-nicas_network_1-1_run
 29/159 Test #333: test_bump_hdiag-nicas_network_1-1_run .......................   Passed    4.02 sec
        Start 334: test_bump_hdiag-nicas_network_1-1_compare
 30/159 Test #334: test_bump_hdiag-nicas_network_1-1_compare ...................   Passed    0.28 sec
        Start 335: test_bump_lct-nicas_one_scale_1-1_run
 31/159 Test #335: test_bump_lct-nicas_one_scale_1-1_run .......................   Passed   12.89 sec
        Start 336: test_bump_lct-nicas_one_scale_1-1_compare
 32/159 Test #336: test_bump_lct-nicas_one_scale_1-1_compare ...................   Passed    0.40 sec
        Start 337: test_bump_nicas_fast_sampling_1-1_run
 33/159 Test #337: test_bump_nicas_fast_sampling_1-1_run .......................   Passed    4.57 sec
        Start 338: test_bump_nicas_fast_sampling_1-1_compare
 34/159 Test #338: test_bump_nicas_fast_sampling_1-1_compare ...................   Passed    0.31 sec
        Start 339: test_bump_nicas_subsamp_hvh_1-1_run
 35/159 Test #339: test_bump_nicas_subsamp_hvh_1-1_run .........................   Passed    6.95 sec
        Start 340: test_bump_nicas_subsamp_hvh_1-1_compare
 36/159 Test #340: test_bump_nicas_subsamp_hvh_1-1_compare .....................   Passed    0.34 sec
        Start 341: test_bump_read_cmat_1-1_run
 37/159 Test #341: test_bump_read_cmat_1-1_run .................................   Passed    5.52 sec
        Start 342: test_bump_read_cmat_1-1_compare
 38/159 Test #342: test_bump_read_cmat_1-1_compare .............................   Passed    0.32 sec
        Start 343: test_bump_read_cmat_serial_1-1_run
 39/159 Test #343: test_bump_read_cmat_serial_1-1_run ..........................   Passed    1.44 sec
        Start 344: test_bump_read_cmat_serial_1-1_compare
 40/159 Test #344: test_bump_read_cmat_serial_1-1_compare ......................   Passed    0.29 sec
        Start 345: test_bump_read_mom_1-1_run
 41/159 Test #345: test_bump_read_mom_1-1_run ..................................   Passed    1.36 sec
        Start 346: test_bump_read_mom_1-1_compare
 42/159 Test #346: test_bump_read_mom_1-1_compare ..............................   Passed    0.09 sec
        Start 347: test_bump_read_nicas_1-1_run
 43/159 Test #347: test_bump_read_nicas_1-1_run ................................   Passed    4.11 sec
        Start 348: test_bump_read_nicas_1-1_compare
 44/159 Test #348: test_bump_read_nicas_1-1_compare ............................   Passed    0.09 sec
        Start 349: test_bump_read_sampling_1-1_run
 45/159 Test #349: test_bump_read_sampling_1-1_run .............................   Passed    3.09 sec
        Start 350: test_bump_read_sampling_1-1_compare
 46/159 Test #350: test_bump_read_sampling_1-1_compare .........................   Passed    0.16 sec
        Start 351: test_bump_read_var_1-1_run
 47/159 Test #351: test_bump_read_var_1-1_run ..................................   Passed    0.41 sec
        Start 352: test_bump_read_var_1-1_compare
 48/159 Test #352: test_bump_read_var_1-1_compare ..............................   Passed    0.02 sec
        Start 353: test_bump_read_vbal_1-1_run
 49/159 Test #353: test_bump_read_vbal_1-1_run .................................   Passed    1.78 sec
        Start 354: test_bump_read_vbal_1-1_compare
 50/159 Test #354: test_bump_read_vbal_1-1_compare .............................   Passed    0.08 sec
        Start 355: test_bump_var_forced_var_1-1_run
 51/159 Test #355: test_bump_var_forced_var_1-1_run ............................   Passed    1.80 sec
        Start 356: test_bump_var_forced_var_1-1_compare
 52/159 Test #356: test_bump_var_forced_var_1-1_compare ........................   Passed    0.09 sec
        Start 357: test_bump_vbal_1-1_run
 53/159 Test #357: test_bump_vbal_1-1_run ......................................   Passed    4.74 sec
        Start 358: test_bump_vbal_1-1_compare
 54/159 Test #358: test_bump_vbal_1-1_compare ..................................   Passed    0.20 sec
        Start 359: test_bump_write_cmat_1-1_run
 55/159 Test #359: test_bump_write_cmat_1-1_run ................................   Passed   12.49 sec
        Start 360: test_bump_write_cmat_1-1_compare
 56/159 Test #360: test_bump_write_cmat_1-1_compare ............................   Passed    0.51 sec
        Start 361: test_bump_write_cmat_serial_1-1_run
 57/159 Test #361: test_bump_write_cmat_serial_1-1_run .........................   Passed    2.86 sec
        Start 362: test_bump_write_cmat_serial_1-1_compare
 58/159 Test #362: test_bump_write_cmat_serial_1-1_compare .....................   Passed    0.48 sec
        Start 363: test_bump_write_mom_1-1_run
 59/159 Test #363: test_bump_write_mom_1-1_run .................................   Passed    1.40 sec
        Start 364: test_bump_write_mom_1-1_compare
 60/159 Test #364: test_bump_write_mom_1-1_compare .............................   Passed    0.17 sec
        Start 365: test_bump_write_nicas_1-1_run
 61/159 Test #365: test_bump_write_nicas_1-1_run ...............................   Passed    5.89 sec
        Start 366: test_bump_write_nicas_1-1_compare
 62/159 Test #366: test_bump_write_nicas_1-1_compare ...........................   Passed    0.38 sec
        Start 367: test_bump_write_sampling_1-1_run
 63/159 Test #367: test_bump_write_sampling_1-1_run ............................   Passed    6.40 sec
        Start 368: test_bump_write_sampling_1-1_compare
 64/159 Test #368: test_bump_write_sampling_1-1_compare ........................   Passed    0.21 sec
        Start 369: test_bump_write_var_1-1_run
 65/159 Test #369: test_bump_write_var_1-1_run .................................   Passed    2.81 sec
        Start 370: test_bump_write_var_1-1_compare
 66/159 Test #370: test_bump_write_var_1-1_compare .............................   Passed    0.09 sec
        Start 371: test_bump_write_vbal_1-1_run
 67/159 Test #371: test_bump_write_vbal_1-1_run ................................   Passed    4.77 sec
        Start 372: test_bump_write_vbal_1-1_compare
 68/159 Test #372: test_bump_write_vbal_1-1_compare ............................   Passed    0.19 sec
        Start 373: test_bump_hdiag-nicas_use_cgal_1-1_run
 69/159 Test #373: test_bump_hdiag-nicas_use_cgal_1-1_run ......................   Passed   12.79 sec
        Start 374: test_bump_hdiag-nicas_use_cgal_1-1_compare
 70/159 Test #374: test_bump_hdiag-nicas_use_cgal_1-1_compare ..................   Passed    0.42 sec
        Start 375: test_bump_normality_2-1_run
 71/159 Test #375: test_bump_normality_2-1_run .................................   Passed    3.96 sec
        Start 376: test_bump_normality_2-1_compare
 72/159 Test #376: test_bump_normality_2-1_compare .............................   Passed    0.19 sec
        Start 377: test_bump_hdiag_cor_diag_all_2-1_run
 73/159 Test #377: test_bump_hdiag_cor_diag_all_2-1_run ........................   Passed    1.26 sec
        Start 378: test_bump_hdiag_cor_diag_all_2-1_compare
 74/159 Test #378: test_bump_hdiag_cor_diag_all_2-1_compare ....................   Passed    0.12 sec
        Start 379: test_bump_hdiag_grid_2-1_run
 75/159 Test #379: test_bump_hdiag_grid_2-1_run ................................   Passed    4.67 sec
        Start 380: test_bump_hdiag_grid_2-1_compare
 76/159 Test #380: test_bump_hdiag_grid_2-1_compare ............................   Passed    0.16 sec
        Start 381: test_bump_hdiag_hyb-rnd_common_2-1_run
 77/159 Test #381: test_bump_hdiag_hyb-rnd_common_2-1_run ......................   Passed    1.39 sec
        Start 382: test_bump_hdiag_hyb-rnd_common_2-1_compare
 78/159 Test #382: test_bump_hdiag_hyb-rnd_common_2-1_compare ..................   Passed    0.18 sec
        Start 383: test_bump_hdiag_mask_stddev_upper_2-1_run
 79/159 Test #383: test_bump_hdiag_mask_stddev_upper_2-1_run ...................   Passed    1.28 sec
        Start 384: test_bump_hdiag_mask_stddev_upper_2-1_compare
 80/159 Test #384: test_bump_hdiag_mask_stddev_upper_2-1_compare ...............   Passed    0.09 sec
        Start 385: test_bump_hdiag-nicas_cor_specific_univariate_2-1_run
 81/159 Test #385: test_bump_hdiag-nicas_cor_specific_univariate_2-1_run .......   Passed    6.68 sec
        Start 386: test_bump_hdiag-nicas_cor_specific_univariate_2-1_compare
 82/159 Test #386: test_bump_hdiag-nicas_cor_specific_univariate_2-1_compare ...   Passed    0.56 sec
        Start 387: test_bump_hdiag-nicas_local_diag_2-1_run
 83/159 Test #387: test_bump_hdiag-nicas_local_diag_2-1_run ....................   Passed   13.57 sec
        Start 388: test_bump_hdiag-nicas_local_diag_2-1_compare
 84/159 Test #388: test_bump_hdiag-nicas_local_diag_2-1_compare ................   Passed    0.54 sec
        Start 389: test_bump_hdiag-nicas_loc_common_2-1_run
 85/159 Test #389: test_bump_hdiag-nicas_loc_common_2-1_run ....................   Passed    6.36 sec
        Start 390: test_bump_hdiag-nicas_loc_common_2-1_compare
 86/159 Test #390: test_bump_hdiag-nicas_loc_common_2-1_compare ................   Passed    0.43 sec
        Start 391: test_bump_hdiag-nicas_mask_check_2-1_run
 87/159 Test #391: test_bump_hdiag-nicas_mask_check_2-1_run ....................   Passed    5.28 sec
        Start 392: test_bump_hdiag-nicas_mask_check_2-1_compare
 88/159 Test #392: test_bump_hdiag-nicas_mask_check_2-1_compare ................   Passed    0.38 sec
        Start 393: test_bump_hdiag-nicas_mesh_based_2-1_run
 89/159 Test #393: test_bump_hdiag-nicas_mesh_based_2-1_run ....................   Passed    6.90 sec
        Start 394: test_bump_hdiag-nicas_mesh_based_2-1_compare
 90/159 Test #394: test_bump_hdiag-nicas_mesh_based_2-1_compare ................   Passed    0.56 sec
        Start 395: test_bump_hdiag-nicas_network_2-1_run
 91/159 Test #395: test_bump_hdiag-nicas_network_2-1_run .......................   Passed    4.58 sec
        Start 396: test_bump_hdiag-nicas_network_2-1_compare
 92/159 Test #396: test_bump_hdiag-nicas_network_2-1_compare ...................   Passed    0.37 sec
        Start 397: test_bump_lct-nicas_one_scale_2-1_run
 93/159 Test #397: test_bump_lct-nicas_one_scale_2-1_run .......................   Passed   19.36 sec
        Start 398: test_bump_lct-nicas_one_scale_2-1_compare
 94/159 Test #398: test_bump_lct-nicas_one_scale_2-1_compare ...................   Passed    0.56 sec
        Start 399: test_bump_nicas_fast_sampling_2-1_run
 95/159 Test #399: test_bump_nicas_fast_sampling_2-1_run .......................   Passed    5.74 sec
        Start 400: test_bump_nicas_fast_sampling_2-1_compare
 96/159 Test #400: test_bump_nicas_fast_sampling_2-1_compare ...................   Passed    0.50 sec
        Start 401: test_bump_nicas_subsamp_hvh_2-1_run
 97/159 Test #401: test_bump_nicas_subsamp_hvh_2-1_run .........................   Passed    8.61 sec
        Start 402: test_bump_nicas_subsamp_hvh_2-1_compare
 98/159 Test #402: test_bump_nicas_subsamp_hvh_2-1_compare .....................   Passed    0.53 sec
        Start 403: test_bump_read_cmat_2-1_run
 99/159 Test #403: test_bump_read_cmat_2-1_run .................................   Passed    5.34 sec
        Start 404: test_bump_read_cmat_2-1_compare
100/159 Test #404: test_bump_read_cmat_2-1_compare .............................   Passed    0.50 sec
        Start 405: test_bump_read_cmat_serial_2-1_run
101/159 Test #405: test_bump_read_cmat_serial_2-1_run ..........................   Passed    1.16 sec
        Start 406: test_bump_read_cmat_serial_2-1_compare
102/159 Test #406: test_bump_read_cmat_serial_2-1_compare ......................   Passed    0.48 sec
        Start 407: test_bump_read_mom_2-1_run
103/159 Test #407: test_bump_read_mom_2-1_run ..................................   Passed    1.26 sec
        Start 408: test_bump_read_mom_2-1_compare
104/159 Test #408: test_bump_read_mom_2-1_compare ..............................   Passed    0.09 sec
        Start 409: test_bump_read_nicas_2-1_run
105/159 Test #409: test_bump_read_nicas_2-1_run ................................   Passed    5.01 sec
        Start 410: test_bump_read_nicas_2-1_compare
106/159 Test #410: test_bump_read_nicas_2-1_compare ............................   Passed    0.10 sec
        Start 411: test_bump_read_sampling_2-1_run
107/159 Test #411: test_bump_read_sampling_2-1_run .............................   Passed    4.42 sec
        Start 412: test_bump_read_sampling_2-1_compare
108/159 Test #412: test_bump_read_sampling_2-1_compare .........................   Passed    0.16 sec
        Start 413: test_bump_read_var_2-1_run
109/159 Test #413: test_bump_read_var_2-1_run ..................................   Passed    0.45 sec
        Start 414: test_bump_read_var_2-1_compare
110/159 Test #414: test_bump_read_var_2-1_compare ..............................   Passed    0.02 sec
        Start 415: test_bump_read_vbal_2-1_run
111/159 Test #415: test_bump_read_vbal_2-1_run .................................   Passed    2.86 sec
        Start 416: test_bump_read_vbal_2-1_compare
112/159 Test #416: test_bump_read_vbal_2-1_compare .............................   Passed    0.09 sec
        Start 417: test_bump_var_forced_var_2-1_run
113/159 Test #417: test_bump_var_forced_var_2-1_run ............................   Passed    2.23 sec
        Start 418: test_bump_var_forced_var_2-1_compare
114/159 Test #418: test_bump_var_forced_var_2-1_compare ........................   Passed    0.08 sec
        Start 419: test_bump_vbal_2-1_run
115/159 Test #419: test_bump_vbal_2-1_run ......................................   Passed    7.17 sec
        Start 420: test_bump_vbal_2-1_compare
116/159 Test #420: test_bump_vbal_2-1_compare ..................................   Passed    0.17 sec
        Start 421: test_bump_write_cmat_2-1_run
117/159 Test #421: test_bump_write_cmat_2-1_run ................................   Passed   26.31 sec
        Start 422: test_bump_write_cmat_2-1_compare
118/159 Test #422: test_bump_write_cmat_2-1_compare ............................   Passed    0.65 sec
        Start 423: test_bump_write_cmat_serial_2-1_run
119/159 Test #423: test_bump_write_cmat_serial_2-1_run .........................   Passed    3.04 sec
        Start 424: test_bump_write_cmat_serial_2-1_compare
120/159 Test #424: test_bump_write_cmat_serial_2-1_compare .....................   Passed    0.66 sec
        Start 425: test_bump_write_mom_2-1_run
121/159 Test #425: test_bump_write_mom_2-1_run .................................   Passed    1.25 sec
        Start 426: test_bump_write_mom_2-1_compare
122/159 Test #426: test_bump_write_mom_2-1_compare .............................   Passed    0.24 sec
        Start 427: test_bump_write_nicas_2-1_run
123/159 Test #427: test_bump_write_nicas_2-1_run ...............................   Passed    5.57 sec
        Start 428: test_bump_write_nicas_2-1_compare
124/159 Test #428: test_bump_write_nicas_2-1_compare ...........................   Passed    0.52 sec
        Start 429: test_bump_write_sampling_2-1_run
125/159 Test #429: test_bump_write_sampling_2-1_run ............................   Passed    8.39 sec
        Start 430: test_bump_write_sampling_2-1_compare
126/159 Test #430: test_bump_write_sampling_2-1_compare ........................   Passed    0.22 sec
        Start 431: test_bump_write_var_2-1_run
127/159 Test #431: test_bump_write_var_2-1_run .................................   Passed    4.61 sec
        Start 432: test_bump_write_var_2-1_compare
128/159 Test #432: test_bump_write_var_2-1_compare .............................   Passed    0.10 sec
        Start 433: test_bump_write_vbal_2-1_run
129/159 Test #433: test_bump_write_vbal_2-1_run ................................   Passed    8.61 sec
        Start 434: test_bump_write_vbal_2-1_compare
130/159 Test #434: test_bump_write_vbal_2-1_compare ............................   Passed    0.19 sec
        Start 435: test_bump_hdiag-nicas_use_cgal_2-1_run
131/159 Test #435: test_bump_hdiag-nicas_use_cgal_2-1_run ......................   Passed   13.65 sec
        Start 436: test_bump_hdiag-nicas_use_cgal_2-1_compare
132/159 Test #436: test_bump_hdiag-nicas_use_cgal_2-1_compare ..................   Passed    0.53 sec
        Start 437: test_bump_read_cmat_parallel-serial_compare
133/159 Test #437: test_bump_read_cmat_parallel-serial_compare .................   Passed    0.05 sec
        Start 438: test_bump_write_cmat_parallel-serial_compare
134/159 Test #438: test_bump_write_cmat_parallel-serial_compare ................   Passed    0.05 sec
        Start 439: test_qg_ens_recenter
135/159 Test #439: test_qg_ens_recenter ........................................   Passed    0.62 sec
        Start 440: test_qg_parameters_bump_cov
136/159 Test #440: test_qg_parameters_bump_cov .................................   Passed    7.85 sec
        Start 441: test_qg_parameters_bump_cov_pairs
137/159 Test #441: test_qg_parameters_bump_cov_pairs ...........................   Passed    7.92 sec
        Start 442: test_qg_parameters_bump_cov_incens
138/159 Test #442: test_qg_parameters_bump_cov_incens ..........................   Passed    7.48 sec
        Start 443: test_qg_parameters_bump_cov_no_ens
139/159 Test #443: test_qg_parameters_bump_cov_no_ens ..........................   Passed    4.13 sec
        Start 444: test_qg_parameters_bump_cov_nsub
140/159 Test #444: test_qg_parameters_bump_cov_nsub ............................   Passed    6.67 sec
        Start 445: test_qg_parameters_bump_hyb
141/159 Test #445: test_qg_parameters_bump_hyb .................................   Passed    4.15 sec
        Start 446: test_qg_parameters_bump_lct
142/159 Test #446: test_qg_parameters_bump_lct .................................   Passed   12.70 sec
        Start 447: test_qg_parameters_bump_loc
143/159 Test #447: test_qg_parameters_bump_loc .................................   Passed    4.48 sec
        Start 449: test_qg_3densvar_bump
144/159 Test #449: test_qg_3densvar_bump .......................................   Passed    0.96 sec
        Start 450: test_qg_3dvar_bump
145/159 Test #450: test_qg_3dvar_bump ..........................................   Passed    0.71 sec
        Start 451: test_qg_3dvar_hybrid_bump
146/159 Test #451: test_qg_3dvar_hybrid_bump ...................................   Passed    0.93 sec
        Start 452: test_qg_4densvar_bump
147/159 Test #452: test_qg_4densvar_bump .......................................   Passed    1.15 sec
        Start 453: test_qg_4dvar_drplanczos_bump
148/159 Test #453: test_qg_4dvar_drplanczos_bump ...............................   Passed    1.70 sec
        Start 454: test_qg_4dvar_drplanczos_hybrid_bump
149/159 Test #454: test_qg_4dvar_drplanczos_hybrid_bump ........................   Passed    2.25 sec
        Start 455: test_qg_dirac_bump_cov
150/159 Test #455: test_qg_dirac_bump_cov ......................................   Passed    0.50 sec
        Start 456: test_qg_dirac_bump_cov_no_ens
151/159 Test #456: test_qg_dirac_bump_cov_no_ens ...............................   Passed    0.41 sec
        Start 457: test_qg_dirac_bump_hyb_field
152/159 Test #457: test_qg_dirac_bump_hyb_field ................................   Passed    0.56 sec
        Start 458: test_qg_dirac_bump_hyb_value
153/159 Test #458: test_qg_dirac_bump_hyb_value ................................   Passed    0.57 sec
        Start 459: test_qg_dirac_bump_lct
154/159 Test #459: test_qg_dirac_bump_lct ......................................   Passed    0.43 sec
        Start 460: test_qg_dirac_bump_loc_3d
155/159 Test #460: test_qg_dirac_bump_loc_3d ...................................   Passed    0.48 sec
        Start 461: test_qg_dirac_bump_loc_4d
156/159 Test #461: test_qg_dirac_bump_loc_4d ...................................   Passed    0.72 sec
        Start 462: test_qg_dirac_id_cov
157/159 Test #462: test_qg_dirac_id_cov ........................................   Passed    0.46 sec
        Start 463: test_qg_dirac_id_loc_3d
158/159 Test #463: test_qg_dirac_id_loc_3d .....................................   Passed    0.39 sec
        Start 464: test_qg_dirac_id_loc_4d
159/159 Test #464: test_qg_dirac_id_loc_4d .....................................   Passed    0.58 sec

100% tests passed, 0 tests failed out of 159

Label Time Summary:
mpi       = 341.43 sec*proc (62 tests)
openmp    = 410.21 sec*proc (87 tests)
saber     = 442.05 sec*proc (159 tests)
script    = 442.05 sec*proc (159 tests)

Total Test time (real) = 442.80 sec

As I say, I’m able to resolve the broken tests locally with the manual fix, but replacing the reference to dashrepo.ucar.edu with gdex.ucar.edu in the pulled packages wasn’t enough by itself to be a total fix. I am hoping to see if folks more familiar with this can suggest where to find the erroneous slash that still causes the bug.

10

Hi @cgrudz,

Thanks for the information. Can you please tell me which version the code are you using? In the latest version v1.2.0 a different URL is used and it should work. I’m checking the URL in v1.1.1 and I can download the files using that one as well. My guess is that perhaps there is a mismatch between versions?
v1.1.1 database: Download Files: JEDI Test and Application files
v1.2.0 database: Download Files: JEDI-MPAS Test and Application Files

Also could you copy messages you get when you run ecbuild for saber? There should be more info in there.

11

@cgrudz I think I know what’s going on. CMakeLists.txt in fv3-bundle 1.1.2 is not pointing to the latest tags of core repos. Please change the tags to the latests tags:
oops 1.2.0
saber 1.2.0
ioda 2.1.0
ufo 1.2.0

We will update CMakeLists.txt asap.

1 Like
12

I might have run this from the container

jedi-tutorial_latest.sif

instead of

jedi-gnu-openmpi-dev_latest.sif

accidentally, the version of the build was “1.1.0”, I’ll double check the build again and go through your changes to the tags above.

1 Like
13

Following up:

  • I did a fresh pull of the dev container and pulled the fv3-bundle from GitHub.
  • I verified, the version the dev container pulls is actually 1.1.0, so I think the issue was not that I used the tutorial container.
  • I changed the version tags in the CMakeLists.txt in fv3-bundle as specified above (see screenshot)
    Screen Shot 2022-02-07 at 10.54.21 AM
    Screen Shot 2022-02-07 at 10.54.21 AM1857×2094 263 KB

However, I get the new error in the build step:

[ 92%] Building CXX object ufo/tools/new_obsop/test/CMakeFiles/test_ufo_opr_autogenerated.dir/__/__/__/test/mains/TestObsOperator.cc.o
[ 92%] Building CXX object ufo/tools/new_qc/test/CMakeFiles/test_autogeneratedfilter.dir/TestAutogeneratedFilter.cc.o
[ 93%] Building CXX object fv3-jedi/src/fv3jedi/CMakeFiles/fv3jedi.dir/Increment/Increment.cc.o
[ 93%] Building CXX object fv3-jedi/src/fv3jedi/CMakeFiles/fv3jedi.dir/Model/fv3lm/ModelFV3LM.cc.o
In file included from /worktmp/fv3-bundle/fv3-jedi/src/fv3jedi/Model/fv3lm/ModelFV3LM.cc:17:
/worktmp/fv3-bundle/fv3-jedi/src/fv3jedi/Model/fv3lm/ModelFV3LM.h:14:10: fatal error: oops/base/ModelBase.h: No such file or directory
   14 | #include "oops/base/ModelBase.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [fv3-jedi/src/fv3jedi/CMakeFiles/fv3jedi.dir/build.make:435: fv3-jedi/src/fv3jedi/CMakeFiles/fv3jedi.dir/Model/fv3lm/ModelFV3LM.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:24793: fv3-jedi/src/fv3jedi/CMakeFiles/fv3jedi.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 93%] Linking CXX executable test_ufo_opr_autogenerated
[ 93%] Built target test_ufo_opr_autogenerated
[ 93%] Linking CXX executable test_autogeneratedfilter
[ 93%] Built target test_autogeneratedfilter
make: *** [Makefile:166: all] Error 2

Could this be related to the version difference in the dev container pull, where fv3-bundle 1.1.2 should be pulled instead?

14

@cgrudz I made a mistake yesterday about the version numbers. There was actually a bug in saber repository and I issued a fix for it. Can you please try building fv3-bundle tag 1.1.3? I updated the saber tag in the CMakeLists.txt there and fixed the bug in saber tag 1.1.2.
Sorry about the confusion and thanks for pointing out the bug.

1 Like
15

Definitely, I’m happy to help out with this – this is helping me learn the system better for my work too. I have a question though, when I clone fv3-bundle from the dev container, I get v.1.1.0. Can I change this with the tag number in the line

# Create project
project( fv3-bundle VERSION 1.1.0 LANGUAGES C CXX Fortran )

of the CMakeList.txt file?

One related question, how do I keep current with bug-fixes? Do I need to be on the code tree to see all the bug fixes as they come out, i.e. am I rebuilding the code within the container as the tree is updated? Thanks!

16

When you clone jcsda/fv3-bundle and go into fv3-bundle directory it will be in master branch. Then you can checkout 1.1.3. In this tag I updated saber tag. changing project( fv3-bundle VERSION 1.1.0 LANGUAGES C CXX Fortran ) will not change the fv3-bundle version that you cloned. Does this answer your question?

After testing I will change the latest version of fv3-bundle to the new tag (from 1.1.2 to 1.1.3). We suggest you always use the latest tag to keep up-to-date with the bug fixes.
I believe in the dev container (jedi-gnu-openmpi-dev_latest.sif) we don’t include the fv3-bundle code. Right?

1 Like
17

Ah, OK, I think I had a misunderstanding about the role of cloning the fv3-bundle while in the container – following the steps to clone in the tutorial I thought that being in the container specified a version of the bundle for the container to pull. I’ll checkout 1.1.3, re-run the build / test, and follow up. Thanks!

1 Like
18

Great, checking out 1.1.3 development branch, build and test all worked on my end, thanks for the support @maryamao!

1 Like