Hello, I’m Zong-De Xie, a graduate student in NCU, Taiwan.
I’m building and testing JEDI-MPAS system for my lab recently. On a 3DVar assimilation test, I encountered an unclear error.
At first, I built up the JEDI-MPAS system and done the tutorial tests on the NCHC of Taiwan’s supercomputer. Note that on this device, sending execution commands in .sh file to nodes is required. No a single error appeared during the tests.
But when I executed a test using grid and data that I’m interested in, I executed the command
“singularity -B /(path) /(path)/*.sif mpiexec -n 1 /(path)/mpasjedi_variational.x 3dvar_bumpcov.yaml”
Then I got the error message:
“[cpn3110:44887] [[6104,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 501
[cpn3110:44887] [[6104,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 501
[cpn3110:44887] [[6104,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 507
[warn] Epoll MOD(1) on fd 23 failed. Old events were 6; read change was 0 (none); write change was 2 (del): Bad file descriptor
[warn] Epoll MOD(4) on fd 23 failed. Old events were 6; read change was 2 (del); write change was 0 (none): Bad file descriptor
srun: error: cpn3110: task 0: Exited with exit code 1”
I have no clue about exactly where the problem is, maybe with bad yaml coding, error within bufr file, bad .sh file configuration, or somehow the JEDI code itself? If more information is needed to solve this, please tell me.
When running 3DVar, MPAS-JEDI uses a static background covariance that is built from BUMP capabilities. BUMP expects various inputs (fields of variance, length scale and regression coefficients) to be supplied in files, and those inputs are specific to details of the application such as the MPAS mesh.
While code for 3DVar is available in the develop branch of various repos under JCSDA · GitHub, we don’t consider 3DVar to be officially “released.” In particular, we don’t yet provide tools for users to generate their own BUMP files for 3DVar.
We expect to release a basic 3DVar capability for MPAS-JEDI relatively soon, probably in September 2023. It will at least allow users to run BUMP-based 3DVar on general MPAS meshes using the static covariances from Jung et al 2023.
Sorry for the late reply. I wanna provide some progress, or say failed tries, and discuss the situation further.
I chose 3DVar to conduct the test, and hoped it has less part that could go wrong. Just want to clarify that I’m not strongly aimed for utilizing the BUMP error model. But it’s good to know about the coming new functions.
Since the 3DVar is not officially released in JEDI-MPAS, I went back to 3DEnVar, but found out the example uses BUMP error model as well, how could I bypass it, maybe by using MPAS static error?
Finally, the original problem is about that the execute program seems read past the end of .bufr observation files. Note that I did use .bufr files from NCEP GDAS as the observational data source in my .yaml file configuration. With suggestions of my senior co-workers, I tried to convert the .bufr files into .nc4 format, but the same error came up again even with the .nc4 files. Maybe the .bufr files are what cause the error? What would you suggest to avoid this problem?
If more detail is needed to evaluate the problem, I’m glad to provide it.
For a small clarification, your error message is related to a file reading “buffer”, not a “.bufr” file. Thus it may not have to do with observation file reading at all, unless you have a traceback that says so. Converting .bufr observations files to .nc4 using ioda-converters is the recommended approach (as far as I know, but I am an external developer now). You could use an identity B as the static B to start, which would eliminate BUMP as a possible source of the error.
Depending on your code version…
See MPASstatic in those files.
However, eventually you will want to generate either the static B or the localization (for ensemble B) with BUMP for a real-data application.