ParSWMS - parallelized version of a simplified three-dimensional computational module of HYDRUS
Compatibility of the HYDRUS (2D/3D) GUI with ParSWMS
Three-dimensional applications often require a large number of finite elements to discretize realistically large transport domains. Even with the fast personal computers currently available, it is virtually impossible to solve within a reasonable computational time problems having more than about half a million nodes or more. To decrease the required computational time, Hardelauf et al. (2007) parallelized SWMS_3D to develop ParSWMS that distributes problems with a large number of elements over multiple processors working in parallel. The ParSWMS code was developed for the LINUX or UNIX workstations using the installed free-wares MPI, PETSc and PARMETIS. Hardelauf et al. (2007) demonstrated that doubling the number of processors may decrease the computational time by up to nearly 50%.
The SWMS_3D model (Šimůnek et al., 1995) was a direct extension of the SWMS_2D code (version 1.2; Šimůnek et al., 1995; direct predecessor of HYDRUS-2D) to three-dimensional flow and transport problems. The model uses the finite element method with tetrahedral linear finite elements to solve the Richards equation for water flow and the advection-dispersion equation with linear sorption for solute transport in three-dimensional transport domains. As such ParSWMS does not consider all processes (e.g., nonlinear and nonequilibrium solute transport, hysteresis of soil hydraulic properties, heat transport) that are considered by HYDRUS (2D/3D) and represents its simplified version.
- Hardelauf, H., M. Javaux, M. Herbst, S. Gottschalk, R. Kasteel, J. Vanderborght, and H. Vereecken, PARSWMS: a parallelized model for simulating 3-D water flow and solute transport in variably saturated soils, Vadose Zone Journal, 6(2), 255-259, 2007.
- Šejna, M., and J. Šimůnek, HYDRUS (2D/3D): Graphical User Interface for the HYDRUS Software Package Simulating Two- and Three-Dimensional Movement of Water, Heat, and Multiple Solutes in Variably-Saturated Media, published online at www.-pc-progress.cz, PC-Progress, Prague, Czech Republic, 2007.
- Šimůnek, J., T. Vogel, and M. Th. van Genuchten, The SWMS_2D code for simulating water flow and solute transport in two-dimensional variably saturated media, Version 1.2, Research Report No. 132, U.S. Salinity Laboratory, USDA, ARS, Riverside, California, 196 pp., 1994. (Download PDF, 12Mb)
- Šimůnek, J., K. Huang, and M. Th. van Genuchten, The SWMS_3D code for simulating water flow and solute transport in three-dimensional variably saturated media, Version 1.0, Research Report No. 139, U.S. Salinity Laboratory, USDA, ARS, Riverside, California, 155 pp., 1995. (Download PDF, 2.7Mb)
- Šimůnek, J., M. Th. van Genuchten, and M. Šejna, The HYDRUS Software Package for Simulating Two- and Three-Dimensional Movement of Water, Heat, and Multiple Solutes in Variably-Saturated Media, Technical Manual, Version 1.0, PC Progress, Prague, Czech Republic, pp. 241., 2006. (PDF 3.1MB)
Implementation of the ParSWMS support into HYDRUS
Since ParSWMS uses input and output files that have different formatting than files for HYDRUS, two new commands were implemented into the HYDRUS (2D/3D) GUI. The first command, i.e., “Export Data to ParSWMS” (called using the menu command “File->Import and Export-> Export Data to ParSWMS”), creates input files (Selector.in and Grid.in) required by ParSWMS from a HYDRUS project, while checking which processes are supported by ParSWMS. The second command, i.e., “Import Data from ParSWMS” (called using the menu command “File->Import and Export-> Import Data from ParSWMS”), reads output files created by ParSWMS and converts them into the HYDRUS (2D/3D) output files.
Newly created input files for ParSWMS (during the “Export Data to ParSWMS” process) are written into the ParSWMS subfolder located in the HYDRUS project working directory (that should be made permanent), from which they can be copied to a required location on a parallelized computer. ParSWMS output files (should be copied into this subfolder (ParSWMS) after calculations are carried out so that the “Import Data from ParSWMS” command can be carried out.
Since the ParSWMS program runs on multiple processors (j – number of processors), each processor creates its own output files for each variable (i.e., pressure head h, water content th, velocities v_x, and concentrations conc) and for nodes that are solved on a particular processor. These output files are called, for example, h.0000j.out (where j=0, …, number of processors). The format is as follows (nodal number n, z, y, a z coordinates, and a particular variable):
Time *** 0 ***
n x(n) y(n) z(n) h(n)
4006 8.00000e+01 2.00000e+01 2.50000e+01 2.38500e+00
4040 9.00000e+01 2.50000e+01 2.50000e+01 2.30800e+00
4074 1.00000e+02 3.00000e+01 2.50000e+01 2.23100e+00
4174 1.10000e+02 4.50000e+01 2.50000e+01 2.15400e+00
. . . . . . . . .
12128 1.20000e+02 5.00000e+01 0.00000e+00 2.70800e+01
12129 1.30000e+02 5.00000e+01 0.00000e+00 2.70000e+01
Time *** 2.5000e-01 ***
n x(n) y(n) z(n) h(n)
4006 8.00000e+01 2.00000e+01 2.50000e+01 2.37142e+00
4040 9.00000e+01 2.50000e+01 2.50000e+01 2.28621e+00
Similarly, files for other variables are called th.0000j.out, conc.0000j.out, vx.0000j.out, vy.0000j.out, a vz.0000j.out.
The process of conversion is relatively complex to insure that it is applicable also for large problems with millions of nodes (nNodes) and hundreds (thousands) of print time layers (nPrint) without breaching memory limits. The output files for a particular variable and a particular processor (e.g., h.0000j.out) are first divided into temporary smaller files, each containing information for one particular print time layer (iPrint). These smaller files for all processor and a single print time layer are then copied (merged) into the main file (e.g., h.out) one time layer at a time. These temporary files are then deleted. Doing this, the program does not need to read multiple (nNodes) potentially large output files simultaneously. ParSWMS also creates multiple output files with information for observation nodes (ObsNod.0000j.out), depending on whether a particular observation node is located in the transport domain solved by a particular processor. Information from these files is also merged into a single output file ObsNode.out.
Other output files with additional output information (i.e., CumQ.out – cumulative water fluxes across boundaries, v_mean.out - actual water fluxes across boundaries, hMean.out – average pressure heads on boundaries, Solute.out – solute fluxes, and Run_inf.out – run time information) are also converted to have a compatible format with HYDRUS and can be thus read and displayed by its GUI.
After the file conversion (using the “Import Data from ParSWMS” command), ParSWMS output can be viewed using the HYDRUS (2D/3D) GUI, since its output files were fully converted into the HYDRUS output file format.
Testing of the ParSWMS support in HYDRUS
We tested this option on the Test8.h3D project, which can be downloaded here H3D_Test8.zip (10MB). Note, that this project has results in permanent directory H3D_Test8 and ParSWMS input/output files are in H3D_Test8\ParSWMS subdirectory.
You can test the new import/export functions in the following way:
- Download the example H3D_Test8.zip and extract files from the archive to a directory with HYDRUS projects
- Open Test8.h3d in the new version of HYDRUS
- Run the command “File->Import data from ParSWMS”
How to use HYDRUS & ParSWMS in other projects:
- All projects intended for ParSWMS import/export must have a „permanent working directory“
- Run the command „Export Data To ParSWMS“ to prepare input files for the ParSWMS calculations. ParSWMS input files are created in the ParSWMS subdirectory
- Run ParSWMS calculations at a supercomputer or Linux cluster and then copy output files back to the ParSWMS subdirectory
- Run the command „Import Data from ParSWMS“ to convert ParSWMS output files to standard HYDRUS output files
- Graphical display of results should be possible after the import
Source code for the parallelized SWMS_3D can be downloaded here ParSWMS.zip (77kB).