Python path when ecbuilding fv3-bundle

Hi, I am testing JEDI FV3 at TACC stampede2 and I want to know how to specify python version or path when ecbuilding JEDI FV3 (fv3-bundle).

TACC stampede2 has a python2 executable at ‘/usr/bin’ and thus, it seems to locate this python2 environment when ecbuiliding JEDI FV3. This makes me not to download test_case data since ‘urllib’ library is not included in default. Although I could solve this issue in python2, I want to use python3 library. I think function ‘ecbuild_find_python’ is somewhat related to this but I could not solve this on my own. I tested several environmental variables (e.g) PYTHON_ROOT, PYTHON_ROOT_DIR, …) but it doesn’t seem to work out.

Thanks,

Jun

Hi Jun - When using python with JEDI you should always use python3. The scripts will not work correctly with python2. JEDI does not provide a version of python: this will be provided by the system administrators for each HPC platform. JCSDA does not maintain or support modules on Stampede2 so I do not know what is available there. Often on HPC systems you have to load a module to obtain an up-to-date version of python3 and supporting python packages. Have you checked with the documentation or the system administrators?

Hi, Mark.

Thanks for your answer.

I experienced this with the JEDI first public release after I installed several libraries with jedi-stack in TACC stampede. This can be a different story for the new version though. I will keep in mind that I need to use python3 for JEDI.

Of course, TACC provides several versions of python3 as a module. I loaded python3 module but ecbuild always finds python2 in the first hand, not python3. Thus, I was wondering if any specific variable or setup to locate python3 (explicitly) when ecbuilding JEDI FV3.

Anyway, I guess I need to ask this issue additionally with TACC administrators.

Thanks again.

The jedi scripts should explicitly call python3, for example:
https://github.com/JCSDA-internal/ioda/blob/5568abd72aa0905e1c90fc570a4062354411c970/test/ioda_dash_downloader.py.in#L1

If you see a place where this is not done please let us know - we should fix it

Since I do not have a permission to JCSDA-internal repo, I can’t see what you suggested but this will be the first line (#!/usr/bin/env python3)

Actually this initial question is somewhat related to the file you suggested. So let me explain more. The main issue was about ctest. As you can see, python2 is selected as a default python interpreter when ecbuilding several repositories in JEDI FV3 at stampede2 by now.

I thought the data was downloaded when ecbuilding JEDI FV3 but it actually seems to be done when doing ctest for each repository. Thus, loading module python3 at stampede2 seems to be required when doing ctests. Although I could compile it successfully, is it okay to employ python2 for configuring some repositories in JEDI FV3?

I also add more about test_case data. I have compiled JEDI FV3 1.0.1 and 1.1.2, respectively within the JCSDA containers (using revert and latest ones) yesterday. When I did ctest for two JEDI FV3s, both have failures even though I tested them in a container (1.0.1 ; about 200, 1.1.2; 600+). My colleague also had the same ctest results. I thought the main reason is data availability of test_case.

It seems to me that the test data of JEDI FV3 v.1.1.x is downloaded in the case of JEDI FV3 v.1.0.1. Since the script can’t locate files and thus, it will download default data files (this may be for v.1.1.x) .

As you know, the data format of UFO and IODA has a difference between two JEDI version, so this brought some failures when I did ctest in JEDI FV3 v1.0.1. It is still the same today.

In the case of JEDI FV3 v1.1.x, I found test_case data wasn’t downloaded yesterday but it is working fine today. Thus, ctest in JEDI FV3 1.1.2 in a container has no failure today. I think the test_data server sometimes doesn’t work well (traffic limit?) but is there anyone who ask about this?

Thanks,

Jun

Hi Jun - sorry - this was the line I was pointing to as an example:

Regarding python in fv3-bundle, let me clarify. If I recall correctly, there are three places where python is used:
(1) in the coding norms tests
(2) in the scripts to download the test data (e.g. get_ioda_test_data)
(3) in the fypp fortran preprocessor used by the saber repository

The ecbuild_find_python() messages you are referring to are from the fckit package. This is only used by item 3 above. Item 2 should be set up to use python3 explicitly. However, it looks like we still may be using /usr/bin/env python in some of the coding norm tests. We can take a look at that if that is giving problems.

Also - the get_* tests to retrieve the test data may have been failing this week (since Monday) because the data server is down for maintenance. It is back up now so you might want to try again.

Hi, Mark.

My responses were delayed because I had to return to the office.
Yeah. I agree with your point. Since I am going to transfer my work to JEDI v1.1.x from JEDI v1.0.0, I checked if python2 is employed in compiling some libraries at JEDI v.1.1.x.

I installed required libraries at stampede2 with jedi-stack and I have found that ‘atlas’ and ‘fckit’ employs python2.

Anyway, they are successfully compiled and I have successfully compiled executables of JEDI v1.1.2 with HPC compilers and libraries in JEDI stack. I also have successfully finished ctests without any errors. Although they seem to be working properly, I will update this thread if any issues happen with my applications.

Oh. JEDI v1.0.x still has issues with ioda and ufo test_case data in doing ctest.
(test_case data at JEDI v1.1.x will be downloaded)
I have tried to finish ctest several times with this JEDI version, but this doesn’t seem to be resolved yet.

Thanks,

Jun

Thanks Jun for the update. I myself am just returning from vacation so I’m just seeing this now. There was some confusion with how the data is hosted on UCAR’s DASH service so that probably explains the v1.0.0 test failures. The code is correct but it’s probably not getting the correct test data. We probably will not prioritize fixing this since a more recent release is available. We recommend that you use the most recent release.