Fix and improve nldd timing and output reporting #6124

jakobtorben · 2025-03-31T10:08:31Z

Improving reporting of NLDD

Changes

This PR fixes some bugs that were present in the Non-Linear Domain Decomposition (NLDD) reporting and improves how statistics from the NLDD solver are collected, displayed, and analyzed.

The key improvements are:

Fixed a bug that the simulation summary reported the NLDD summary for rank 0 only. Now, it only reports the total local solve time in the simulation summary, and more detailed reports are shown in the DBG file. Removing the breakdown from the final summary report is motivated by the fact that this does not make sense for NLDD in parallel, where all the ranks have to wait for the slowest rank. Instead, we now output detailed reports per domain and per rank in the DBG file, which makes more sense.
There were also some inconsistencies in the NLDD assembly reporting, which have been fixed.
Better reporting structure: Changed BlackoilModel::localAccumulatedReports() from returning a simple SimulatorReportSingle to a more comprehensive SimulatorReport that can track both successful and failed solves.
Per-domain statistics: Added a new method BlackoilModel::domainAccumulatedReports() that provides detailed statistics for each individual domain on the current rank.
Cell-level visualization: Implemented BlackoilModel::writeNonlinearIterationsPerCell() to generate ResInsight-compatible files showing nonlinear iteration counts at the cell level.
Added a helper method BlackoilModel::hasNlddSolver() that makes it cleaner to check if an NLDD solver exists before attempting to use it.
Created an NLDD-specific report writer with more details, used for rank- and domain-level statistics.

New Features

1. Domain Distribution Summary

The solver now provides a summary of how cells, wells and domains are distributed across ranks:

NLDD domain distribution summary:
  rank   owned cells   overlap cells   total cells   wells   domains
--------------------------------------------------------------------
     0         9682           2227         11909       5       12
     1        11809           1777         13586      14       12
     2        12131            536         12667      12       14
     3        10809            590         11399       5       13
--------------------------------------------------------------------
   sum        44431           5130         49561      36       51

2. Per-Domain Performance Reports

Each domain's performance is now reported with detailed statistics in the DBG file:

======  Accumulated local solve data for domain 0 on rank 1 ======
Owned + overlap cells:          1086
Number of wells:                   2
Number of domains:                 1
-------------------------------------------------------
Total time:                        0.00 s
  Pre/post/wait time:              0.00 s
  Solver time:                     2.87 s (Wasted: 0.1 s; 1.9%)
    Assembly time:                 1.39 s (Wasted: 0.0 s; 1.5%)
      Well assembly:               0.06 s (Wasted: 0.0 s; 1.3%)
    Linear solve time:             0.45 s (Wasted: 0.0 s; 2.3%)
      Linear setup:                0.19 s (Wasted: 0.0 s; 2.3%)
    Props/update time:             1.02 s (Wasted: 0.0 s; 2.2%)
Overall Linearizations:          1250   (Wasted:    22; 1.8%)
Overall Nonlinear Iterations:     787   (Wasted:    21; 2.7%)
Overall Linear Iterations:       1253   (Wasted:    12; 1.0%)
-------------------------------------------------------
Converged domain solves:         462
  Accepted with relaxed tol:       0
Unconverged domain solves:         1

3. Per rank statistics

Detailed NLDD statistics accumulated per rank is now shown per rank in the DBG file (what was former shown in the simulation summary).

======  Accumulated local solve data for rank 1 ======
Owned + overlap cells:         13586
Number of wells:                  14
Number of domains:                12
-------------------------------------------------------
Total time:                       31.76 s
  Pre/post/wait time:              2.10 s
  Solver time:                    29.66 s (Wasted: 0.1 s; 0.2%)
    Assembly time:                13.91 s (Wasted: 0.0 s; 0.1%)
      Well assembly:               0.30 s (Wasted: 0.0 s; 0.2%)
    Linear solve time:             5.35 s (Wasted: 0.0 s; 0.2%)
      Linear setup:                1.92 s (Wasted: 0.0 s; 0.2%)
    Props/update time:            10.22 s (Wasted: 0.0 s; 0.2%)
Overall Linearizations:         14517   (Wasted:    22; 0.2%)
Overall Nonlinear Iterations:    8961   (Wasted:    21; 0.2%)
Overall Linear Iterations:      13370   (Wasted:    12; 0.1%)
-------------------------------------------------------
Converged domain solves:        5555
  Accepted with relaxed tol:       0
Unconverged domain solves:         1

4. Visualisation Capabilities

The PR adds the ability to visualise nonlinear iteration counts:

Cell-level NLDD iteration count: A ResInsight-compatible output file that allows visualisation of nonlinear iteration counts per cell, helping identify problematic regions in the model.

Domain-level performance: The new detailed domain statistics make it possible to analyse the NLDD method in more detail and create useful plots like the one below.

Implementation Details

The implementation extends the existing SimulatorReport structures for domain-specific performance reporting. A dedicated one can be created for NLDD to avoid having some unused fields during the current default Newton method. However, there is quite a lot of functionality built around it for things like adding reports and etc. So, I kept it as one report to avoid too much duplication.
To produce the NLDD statistics there is a significant amount of processing that needs to be done. However, all the processing-intensive and global calls are only done once per simulation, and I argue that the value it provides justifies the cost.
In the NLDD domain statistics report, there are the three following fields:
Total time: Total time spent in NLDD for this domain
Pre/post/wait time: Total time spent preparing for NLDD solve and waiting for other ranks to finish
Solver time: Total time spent doing an actual solve for this domain
The total solve time will therefore be around the same for all domains and about the same as the local solve time in the simulations summary. Whereas the pre/post/wait time will differ based on the workload on that rank.

… summary

jakobtorben · 2025-03-31T10:08:47Z

jenkins build this please

bska

To me this looks good for the most part. I haven't looked very hard into the details, but the parts that I have checked seem sound to me. I personally prefer using std::size_t from <cstddef> over "naked" size_ts (from <stddef.h>). You may consider switching to the std:: variant.

Other than that, I have only minor comments.

bska · 2025-04-01T18:31:39Z

opm/simulators/flow/FlowMain.hpp

+                {
+                    // Write the number of nonlinear iterations per cell to a file in ResInsight compatible format
+                    const auto& odir = eclState().getIOConfig().getOutputDir();
+                    simulator_->model().writeNonlinearIterationsPerCell(odir);
+
+                    // Create a deferred logger and add the report with rank as tag
+                    DeferredLogger local_log;
+                    std::ostringstream ss;
+
+                    // Accumulate reports per domain
+                    const auto& domain_reports = simulator_->model().domainAccumulatedReports();
+                    for (size_t i = 0; i < domain_reports.size(); ++i) {
+                        const auto& dr = domain_reports[i];
+                        ss << "======  Accumulated local solve data for domain " << i << " on rank " << mpi_rank_ << " ======\n";
+                        dr.reportNLDD(ss);
+                        // Use combined rank and domain index as tag to ensure global uniqueness and correct ordering
+                        local_log.debug(fmt::format("R{:05d}D{:05d}", mpi_rank_, i), ss.str());
+                        ss.str(""); // Clear the stringstream
+                        ss.clear(); // Clear any error flags
+                    }
+                    // Gather all logs and output them in sorted order
+                    auto global_log = gatherDeferredLogger(local_log, FlowGenericVanguard::comm());
+                    if (this->output_cout_) {
+                        global_log.logMessages();
+                    }
+                }
+                {
+                    // Create a deferred logger and add the report with rank as tag
+                    DeferredLogger local_log;
+                    std::ostringstream ss;
+                    ss << "======  Accumulated local solve data for rank " << mpi_rank_ << " ======\n";
+                    simulator_->model().localAccumulatedReports().reportNLDD(ss);
+                    // Use rank number as tag to ensure correct ordering
+                    local_log.debug(fmt::format("{:05d}", mpi_rank_), ss.str());
+
+                    // Gather all logs and output them in sorted order
+                    auto global_log = gatherDeferredLogger(local_log, FlowGenericVanguard::comm());
+                    if (this->output_cout_) {
+                        global_log.logMessages();
+                    }
+                }


Maybe we could push this logic to a separate helper function? That way, runSimulatorAfterSim_() wouldn't be dominated by NLDD-related output.

I thought the same thing but never got around to doing it. Added this now and also moved some functions out from BlackOilModelNldd.hpp into this new NlddReporting.hpp file.

bska · 2025-04-01T18:40:10Z

One more general question: If a model has many local domains (e.g. more than 5000), either because the operator requests a very fine-grained partition or because the model "just" has a lot of active cells, how do the statistics plots look then? Do they become very crowded?

jakobtorben · 2025-04-02T14:23:47Z

The use of size_t should be removed.

For the NLDD_ITER plot in ResInsight, I don't think this would be an issue. Here we could also add a filter to only show the cells above a certain value. For the other plot I included, this is just something I made myself based on the domain statistics so here it is up to the user how they want to plot the statistics.

What I was slightly worried about is if we write many domain reports to the DBG file, this would clutter the file or the file would be too large. This is already starting to become a problem with NLDD, which has a lot more output. But I ended up adding it to the DBG file as the logger in OPM conveniently solved all the problems I had for logging and sorting this info. So for now, I think this is fine, but this is something we can consider fixing later.

jakobtorben · 2025-04-02T14:24:02Z

jenkins build this please

bska

Thanks a lot for the improvements here. This is starting to look good to me. I (think I) have identified a couple of minor issues, the most important of which may be a differing function type for the domainAccumulatedReports() function. Is that function called at all?

Other than that, we should use ::Opm:: as the namespace prefix when we need to disambiguate Opm from within the Opm namespace.

bska · 2025-04-02T15:03:52Z

opm/simulators/flow/NlddReporting.hpp

+        if (!resInsightFile) {
+            OPM_THROW(std::runtime_error,
+                      "Failed to open NLDD nonlinear iterations output file: " + fname.string());
+        }


I don't think you really need this check. The std::ofstream constructor will itself throw an exception if the it fails to open the output file.

You are right. And apologies for not fixing this since you pointed this about in the last PR. This code was written before that...

bska · 2025-04-02T15:12:03Z

opm/simulators/flow/NlddReporting.hpp

You should probably add this new file (NlddReporting.hpp) to the set of build files too (top-level CMakeLists_files.cmake file). If the file is supposed to be installed and usable from outside of opm-simulators, and I think it probably should be, then it should go into the PUBLIC_HEADER_FILES. Otherwise, I think it should go into the MAIN_SOURCE_FILES. It is, admittedly, not quite as critical in that case, but it does nevertheless help the build system to generate file listings for those who use IDEs.

Thanks for clarifying what should go into this file! Added it to PUBLIC_HEADER_FILES now.

bska · 2025-04-02T15:14:08Z

opm/simulators/flow/BlackoilModelNldd.hpp

+        domain_reports_accumulated_.resize(num_domains);
+
+        // Print domain distribution summary
+        Opm::printDomainDistributionSummary(


We're already in the Opm namespace, so this Opm:: prefix is a little misleading. If you want it, then Opm:: itself should be prefixed by the global namespace, meaning this function call becomes

::Opm::printDomainDistributionSummary(...);

instead.

Ahh yes I was following the pattern in the partitionCells call but hastly overlooked the prefixed ::, added this prefix now.

bska · 2025-04-02T15:17:33Z

opm/simulators/flow/BlackoilModelNldd.hpp

    {
        return local_reports_accumulated_;
    }

+    /// return the statistics of local solves accumulated for each domain on this rank
+    std::vector<SimulatorReport>& domainAccumulatedReports()


Did you mean to return a reference here? Unless I'm reading it incorrectly, the header file declares this function as returning an object (a vector<SimulatorReport>) rather than a reference to a mutable vector.

Yes, this was intentional, as I don't want to make a copy of the reports, given that the object can be quite large.
There was indeed a mismatch in the function definitions in BlackOilModel.hpp and BlackOilModelNldd.hpp.

The complication here was that domainAccumulatedReports() needed to count the wells for each domain using the mapping found in the well model for all the wells seen so far. So, we need to mutate this internal variable whenever we use this function to ensure this is updated. We could, of course, always keep this count updated, but I didn't want to do extra work since this is only used once in the end. Using the wells at the end from the schedule like we do for the per-rank reports is not possible since we do not have the well to domain mapping available here.

To overcome this issue, I made the functions consistently a const functions but made the domain_reports_accumulated_ member mutable. To also ensure that I returned a reference, I removed returning an empty object if not used by NLDD, and instead raised an error.

jakobtorben · 2025-04-03T07:09:32Z

jenkins build this please

jakobtorben · 2025-04-03T08:10:40Z

jenkins build this please

jakobtorben · 2025-04-03T08:16:14Z

jenkins build this please

jakobtorben · 2025-04-03T08:24:31Z

jenkins build this please

jakobtorben · 2025-04-03T08:55:16Z

Thanks for your comments @bska @akva2 ! This should be ready from my side now.

jakobtorben added 10 commits March 31, 2025 09:05

Stop adding global well assembly time and well controls to local report

5166f09

Add NLDD specific SimulatorReport report writing

0c773a4

Fix and improve timing and iteration count in NLDD

4533a60

Write per rank NLDD summary to DBG instead of only rank 0 NLDD to sim…

140efe7

… summary

Add NLDD domain and well distribution summary

360a33d

Pass local_report as reference to ensure correct converged status update

91d3d5b

Add more output data to NLDD simulation summary

c968518

Improve NLDD reporting with per-domain solve statistics

5d0363a

Log domain data for wells and cells

ba0a45f

Write NLDD nonlinear iterations to file

f421d77

bska reviewed Apr 1, 2025

View reviewed changes

atgeirr added this to the Release 2025.04 milestone Apr 2, 2025

jakobtorben force-pushed the fix_and_improve_nldd_timing_and_output_reporting branch from 19151fd to 9faa3f8 Compare April 2, 2025 14:17

bska reviewed Apr 2, 2025

View reviewed changes

jakobtorben force-pushed the fix_and_improve_nldd_timing_and_output_reporting branch from 9faa3f8 to 90d0ff7 Compare April 3, 2025 07:09

jakobtorben force-pushed the fix_and_improve_nldd_timing_and_output_reporting branch from 90d0ff7 to ddbb1dd Compare April 3, 2025 08:10

jakobtorben force-pushed the fix_and_improve_nldd_timing_and_output_reporting branch from ddbb1dd to bc20dfd Compare April 3, 2025 08:15

jakobtorben force-pushed the fix_and_improve_nldd_timing_and_output_reporting branch from bc20dfd to 35e1f59 Compare April 3, 2025 08:24

Move NLDD reporting implementation out to separate file

35e1f59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix and improve nldd timing and output reporting #6124

Fix and improve nldd timing and output reporting #6124

jakobtorben commented Mar 31, 2025

jakobtorben commented Mar 31, 2025

bska left a comment

bska Apr 1, 2025

jakobtorben Apr 2, 2025

bska commented Apr 1, 2025

jakobtorben commented Apr 2, 2025

jakobtorben commented Apr 2, 2025

bska left a comment

bska Apr 2, 2025

jakobtorben Apr 3, 2025

bska Apr 2, 2025

jakobtorben Apr 3, 2025

bska Apr 2, 2025

jakobtorben Apr 3, 2025

bska Apr 2, 2025

jakobtorben Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

Fix and improve nldd timing and output reporting #6124

Are you sure you want to change the base?

Fix and improve nldd timing and output reporting #6124

Conversation

jakobtorben commented Mar 31, 2025

Improving reporting of NLDD

Changes

New Features

1. Domain Distribution Summary

2. Per-Domain Performance Reports

3. Per rank statistics

4. Visualisation Capabilities

Implementation Details

jakobtorben commented Mar 31, 2025

bska left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bska commented Apr 1, 2025

jakobtorben commented Apr 2, 2025

jakobtorben commented Apr 2, 2025

bska left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025

jakobtorben commented Apr 3, 2025