(inverse) Big data input on Inverse Modeling

A discussion forum for Hydrus-1D users.
infoleon
Posts: 10
Joined: Tue Nov 07, 2017 6:05 pm
Location: Brazil

(inverse) Big data input on Inverse Modeling

Post by infoleon » Wed May 09, 2018 2:01 am

Hi,

I am having problem with inverse modeling and lots of input data. The GUI do not let me create the files through it, because it is too much data. But thats not the problem.

The problem is that when I run the simulation, it gives me the answer exactly as the input. No variation, no difference on 95% confidence limits.

It seems to run, it is taking 1500 secs each run, generating 3.4 Gb output files.
If I reduce the input data (randomly throwing away data), it works normal.

Is it something with my raw data?
Or is it another problem? (like file too big, memory usage or other crazy computer thing).
I am using a program that I made to create the input files, so the formatting and preparation of the files should not be responsible for this difference in results as far as I know.

Thanks a lot!
Leo.

PS:
Here is part of the Fit.out file that I got using all the data:

Code: Select all

...
Mass balance error in FE solution during final run was  0.0000 %


 Correlation matrix
 ==================
         1          2          3          4          5
   1   0.0000
   2   0.0000     0.0000
   3   0.0000     0.0000     0.0000
   4   0.0000     0.0000     0.0000     0.0000
   5   0.0000     0.0000     0.0000     0.0000     0.0000

 Non-linear least-squares analysis: final results
 ================================================
                                           95% Confidence limits
 Variable        Value        S.E.Coeff.     Lower         Upper
   WCR      0.10000E-02    0.41488E-15  0.10000E-02  0.10000E-02
   WCS      0.50000E+00    0.41488E-15  0.50000E+00  0.50000E+00
   ALPHA    0.59000E-01    0.41488E-15  0.59000E-01  0.59000E-01
   N        0.14800E+01    0.41488E-15  0.14800E+01  0.14800E+01
   CONDS    0.13100E+01    0.41488E-15  0.13100E+01  0.13100E+01


 Contributions to the final objective function

 Measurement Set   2:   0.2402E+04

 RSQUARE for regression of predicted vs observed =0.00000

 Mean Weighted Error, ME               :  0.4098E+00
 Mean Weighted Absolute Error, MAE     :  0.4098E+00
 Root Mean Square Weighted Error, RMSE :  0.4148E+00


 Akaike Information Criterion, AIC     : -0.2456E+05
 Bayesian Information Criterion, BIC   : -0.2452E+05
 ...
 
 ...
 Inverse solution calculation time [sec]   1315.65999998152     ---> last line!
...

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Wed May 09, 2018 4:02 pm

It seems that the program is not fitting any of your data (Rsquare=0, correlations=0, etc). Have you looked at our examples (either distributed with the program or posted online)? Have you done the inverse h1D tutorial? J.

infoleon
Posts: 10
Joined: Tue Nov 07, 2017 6:05 pm
Location: Brazil

Re: (inverse) Big data input on Inverse Modeling

Post by infoleon » Wed May 09, 2018 4:30 pm

Have you looked at our examples (either distributed with the program or posted online)?
Ans: Yes. I looked at the Evaporation Experiment with inverse modeling (with Ole Wendroth), similar to my experiment. But not too much more.

Have you done the inverse h1D tutorial?
Ans: No. Which one? I've read the manual, though.

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Wed May 09, 2018 4:47 pm

For example these. J.
Attachments
Inverse modeling.pdf
(362.12 KiB) Downloaded 123 times

Daniel
Posts: 6
Joined: Wed Jun 20, 2018 1:46 pm
Location: Canada

Re: (inverse) Big data input on Inverse Modeling

Post by Daniel » Fri Jun 22, 2018 4:32 pm

Dear Colleagues,
I’ve been working with Hydrus1D for several years, very nice free program and really useful although every now and again it shows some strange error messages and stopped working. This time it seems however, that I did not find a way to overcome this problem, since the error occurred inconsistent and is hard to curtail and to comprehend.
I tried to adjust simulated SWC data to the measured values (rain response) in order to obtain alpha, n (Qs is fixed, Qr could be fixed and Ks could be fixed too from the literature, I basically need alpha and n).
I run 100 time steps (hourly resolution) for several two layer systems, but I am only interested in layer 1 since the SWC sensor is in 5cm (data for inverse solution: x – number of obs; Y – SWC value; Type 2; Position 5; Weight 0). I am interested in the response to several rain events and I have several soil profiles, were only layer 1 changed minor due to different amendments (layer 2 constant). SWC values were about 0.13 to 0.22 for every measured data set. Here is an example of the hydrological settings:

Layer 1 (loamy sand)
Qr Qs alpha n Ks I
0.15 0.44 0.08 2.68 15 0.5 Initial setting
0.06 0 0.04 0 10 0 MIN
0.44 0 0.4 5 30 0 MAX
Yes no yes yes yes no

Layer 2 (sand)
0.13 0.43 0.145 2.68 29.7 0.5
0.1 0 0 0 10 0
0.43 0 2 3 40 0
Yes no yes yes yes no

Hydrus inversion works fine for some data sets but for the majority, the calculations stopped (“Numerical solution has been stopped since it has not converged in 10 consecutive time steps!”) although nothing seems to be wrong about the setting.
I checked all settings (according to the inverse modeling pdf) and they seem correct and I run without problems but like a said only for a limited number of simulations. I read the related comments on this forum but nothing works so far. The error is hard to comprehend. For example, identical settings (e.g. from above) won’t work for different measured data, although the measured SWC curves are very similarl (same data range, format etc.). Even more strange: sometimes, tiny alterations in the hydrological settings will result in a failed simulation as well. For example, the above mentioned two layer soil works as long as I use Qs = 44%, however it won’t run under Qs 43.9% nor under 44.1% and this is really hard to understad and annoying. I stuck here since two days trying some irrational slightly changes in order to obtain reliable results, which sometimes work but most of the time not.

Moreover, I realized, that the results if the simulations are completed are almost identical to the input parameters, e.g alpha 2.68 resulted in 2.67999.
Does anyone of you guys have an idea about how to tackle this problem? I would really appreciate.

I need to automatically adjust the parameters to avoid arbitrarily. Thanks you!
Best Daniel

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Fri Jun 22, 2018 9:32 pm

You probably have errors in the input:
a. Position: this is not the depth, but the observation node number.
b. Weight: it should not be zero. If you put zero weight, the data point is not considered (unless you put zero weight on all data points and then they will be uniformly weighted.)
c. It is better not to place constraints in optimized parameters (unless the code goes into some unreasonable territory).
d. Before you run the optimization, make sure that the code actually converges, i.e., in the first run request only zero inverse iteration. Only once this run works well set the number of inverse iteration higher and do the optimization.
J.

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Fri Jun 22, 2018 9:32 pm

You probably have errors in the input:
a. Position: this is not the depth, but the observation node number.
b. Weight: it should not be zero. If you put zero weight, the data point is not considered (unless you put zero weight on all data points and then they will be uniformly weighted.)
c. It is better not to place constraints in optimized parameters (unless the code goes into some unreasonable territory).
d. Before you run the optimization, make sure that the code actually converges, i.e., in the first run request only zero inverse iteration. Only once this run works well set the number of inverse iteration higher and do the optimization.
J.

Daniel
Posts: 6
Joined: Wed Jun 20, 2018 1:46 pm
Location: Canada

Re: (inverse) Big data input on Inverse Modeling

Post by Daniel » Mon Jun 25, 2018 6:15 pm

Dear Jirka,
Thank you very much for your quick response.
a) I mistake the number of knots with the number of observation points. Now I changed the number to the observation note at 5cm. I also removed all unnecessary knots to be on the safe side
b) I had zero weight on all data points so all measured SWC data should weighed equally as you mentioned. Still this time all data points were changed to 1 (as suggested in the manual)
c) I removed all constrains as suggested (0, 0)
d) I started each simulation with 0 iteration to make sure that the code converges properly
The number of successfully inversions did not increased. For simulations that run through, the values make a bit more sense now since they changed and I am happy to see the progress here. Still the main problems remain:
a) Simulations won’t run for certain data sets although nothing else changed but the measured SWC.
b) Simulations won’t run for certain settings, e.g. if I increase the Qs value from say 0.44 to 0.5 although nothing else changes but Qs (not even the measured SWC data)
c) The simulations run for 0 iterations fine but they won’t run for more than 0 if Hydrus doesn’t ‘like’ the data sets or setting (as mentioned above) or – even for successfully calculated simulations – if I increase the number of iterations > 3 (0 -3 runs fine).
Maybe something in the other settings (time settings) is wrong but I wonder why some data were processed fine and some not. The same is true for the settings. I am still kind of clueless right now and I tried almost everything.
I attached a data set of measured SWC data as well as one of my Hydrus models to give you an idea of what I am dealing with. Main goal is to fit alpha and n for different treatments. Any further recommendation is highly welcome. Thank you!
Best Daniel
Attachments
Inverse_sheet_100h_event_1.xlsx
SWC measured and hydraulic settings
(21.68 KiB) Downloaded 67 times

Daniel
Posts: 6
Joined: Wed Jun 20, 2018 1:46 pm
Location: Canada

Re: (inverse) Big data input on Inverse Modeling

Post by Daniel » Tue Jun 26, 2018 3:04 pm

It appears that for some reason the second attachment was lost, so here we go again, hope its gonna work, its a zip file,
Best Daniel
Attachments
N10_real_Qs.zip
Hydrus 1D example inverse simulations
(68.52 KiB) Downloaded 54 times

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Wed Jun 27, 2018 4:11 am

It has been pretty simple to get a reasonable fit. See below.
Untitled.png
Untitled.png (62.4 KiB) Viewed 1760 times
Please note that I do not think that I will have time to look at your data in the coming days/weeks. I’m teaching this week a HYDRUS short course in Golden, Colorado, and on Saturday I’m leaving for a three-week trip to China (Beijing, Lhasa, Chengdu, Guilin, Hong Kong).

J.

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Wed Jun 27, 2018 4:16 am

The project itself. J.
SUPER_N10_real_Qs.zip
(61.58 KiB) Downloaded 65 times

Daniel
Posts: 6
Joined: Wed Jun 20, 2018 1:46 pm
Location: Canada

Re: (inverse) Big data input on Inverse Modeling

Post by Daniel » Thu Jun 28, 2018 7:50 pm

Dear Jirka,
Thank you so much for your time and your rapid support, after reinstalling Hydrus1D it finally works. I realized that some simulations demand lower iteration numbers (as low as 3-6) to perform error-free operations but this is survivable. However, the Ks values are pretty high (about 10times than expected) unless the units for input and output are different, mm/h instead of cm/h, is this possible? There are no unit given in result file, nor in the setting box, once one aimed for inversion results.
The other issue might stand in the same context: fixing initial Ks to values from the literature (SPAW) instead of estimation causes errors again. So I need to constrain them, but then each inversion hits the (upper) boundary. Unit problem?
Thank you for a quick answer whenever time allows it.
Best Daniel

Jirka
Posts: 4712
Joined: Sat Mar 16, 2002 3:47 pm
Location: USA
Location: Riverside, CA

Re: (inverse) Big data input on Inverse Modeling

Post by Jirka » Thu Jun 28, 2018 8:09 pm

The units that you select are used everywhere. The help gives pretty clear instructions on what "Iteration Criteria" to use. We even state there "We do not recommend to change these default values unless you are an experienced user and know exactly what you are doing!" J.

Daniel
Posts: 6
Joined: Wed Jun 20, 2018 1:46 pm
Location: Canada

Re: (inverse) Big data input on Inverse Modeling

Post by Daniel » Fri Jun 29, 2018 4:05 pm

Thank you very much Jirka, I didn’t change the default settings, so that means the Ks values need to be constrained in our case.
Have a safe journey!
Best Daniel

Daniel
Posts: 6
Joined: Wed Jun 20, 2018 1:46 pm
Location: Canada

Re: (inverse) Big data input on Inverse Modeling

Post by Daniel » Wed Jul 04, 2018 5:31 pm

Dear Colleagues, Dear Jirka,
I still have problems to get my inversions for larger time periods running. Regardless of the number of iterations, it won’t run, unless I used 0 iteration. So far I can’t see any user error in the model or perhaps: larger time periods (< 1500h) aren’t suitable for inversion? To whom it may concern, I attached one of my models. Maybe one off you know why it always stops working. Any suggestion is highly welcome,
Cheers Daniel
Attachments
C_1_full.zip
Hydrus1D example inversion wont run
(28.72 KiB) Downloaded 52 times

Post Reply