Incorporate determinism into GPU heuristics + parallel determinsitic B&B+GPU exploration

[aliceb-nv]

This PR implements the integration of GPU heuristics into the deterministic codepath of the solver. This enables running full end-to-end solves with presolve, parallel B&B, and GPU heuristics, with full run-to-run determinism and heterogeneous (CPU+GPU) execution.

For this purpose, the GPU heuristics were modified to eliminate sources of nondeterminism such as non-ordered floating point reductions, atomic operations, scheduling-dependent operations... Extensive testing has been added to the CI runs to ensure individual components (such as probing, FJ, recombiners, FP...) are bitwise deterministic, and their overall orchestration maintains this invariant.
Furthermore, to reduce changes to the codebase and splits in the codepath, wall-clock timers on the heuristics side were replaced with unified termination checking objects, that check for wall time (opportunistic mode), or work total (deterministic mode).
These timers are hierarchical, and thus allow for checking against multiple terminations sources, for example a local algo's work budget, the global wall clock time limit, or in future PRs user-controlled termination signals like Ctrl-C or callback handlers.

B&B acts as the authority for incumbent publishing to the user. At every horizon sync, the queue of GPU heuristic solutions is checked, and if solutions whose timestamp is <= the current B&B work total, they are drained, replayed, and incorporated into the tree exploration.

A new get_solution_callback_ext callback has been introduced to prevent ABI breaks and pass more metadata with each published solution. For now, only the solution origin and its work unit timestamp are exposed, but additional fields may be later added without breaking the ABI.
The benchmark runner now relies on such callbacks to record the incumbents and their timestamps - it is no longer necessary to parse the logs to compute the primal integral.

A few other bugfixes have been included to the existing B&B deterministic exploration regarding node queues. Other fixes include uninitialized memory accesses, and improvements to allow running compute-sanitizer initcheck without false positives. Determinism logs have also been added thorought the code, disabled by default.

Some features remain unsupported in deterministic mode, such as Papilo's Probing presolver, concurrent root solve, RINS, or the sub-MIP recombiner. These will be incorporated in later PRs.
Furthermore, the PR has been modified to take the ML estimators out for the sake of reducing code review workload. They will be incorporated again in later PRs, along with improved tuning.
No intentional API or ABI breaking changes introduced.

Opportunistic mode performance remains unchanged:
Primal gap 12.5%, 226 feasible, primal integral 19.1%.

Deterministic mode benchmarks pending.

Closes #144
Closes #882

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (6)

cpp/tests/mip/diversity_test.cu (4)

69-69: Empty function should be removed or documented.

setup_device_symbols is a no-op function. If it's a placeholder for future implementation, add a TODO comment; otherwise, remove it to avoid confusion.

💡 Suggested fix

Either remove the function and its call sites, or document the intent:

-static void setup_device_symbols(rmm::cuda_stream_view stream_view) { (void)stream_view; }
+// TODO: Implement device symbol initialization if needed for determinism tests
+// static void setup_device_symbols(rmm::cuda_stream_view stream_view) { (void)stream_view; }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tests/mip/diversity_test.cu` at line 69, The function
setup_device_symbols(rmm::cuda_stream_view stream_view) is a no-op; either
remove this unused function (and clean up any call sites that only exist to
invoke it) or make its purpose explicit by adding a TODO/comment indicating it's
a deliberate placeholder for future device-symbol setup; update references to
setup_device_symbols in tests (e.g., any calls in diversity_test.cu) to reflect
the removal or to clarify that the call is intentionally a no-op.

---

71-73: Unused seed parameter in test runner functions.

The seed parameter in test_full_run_determinism (line 72) and test_initial_solution_determinism (line 121) has a default value but is never used within the function body. The global seed is set externally via cuopt::seed_generator::set_seed(seed) in the calling test. Either remove the parameter or use it to seed something locally.

💡 Suggested fix

If the seed is intended to be set locally within these functions:

 static uint32_t test_full_run_determinism(std::string path,
-                                          unsigned long seed = std::random_device{}(),
+                                          unsigned long seed,
                                           float work_limit   = 10.0f)
 {
+  cuopt::seed_generator::set_seed(seed);
   const raft::handle_t handle_{};

Or remove the parameter if external seeding is intentional.

Also applies to: 120-121

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tests/mip/diversity_test.cu` around lines 71 - 73, The two test runner
functions test_full_run_determinism and test_initial_solution_determinism
declare an unused seed parameter; either remove the seed parameter from both
function signatures (and update any callers) if external seeding is intended, or
make the functions actually use the value by calling
cuopt::seed_generator::set_seed(seed) (or equivalent local seeding) at the start
of each function to apply the provided seed; pick one approach and apply it
consistently to both functions.

---

382-394: Test instantiation includes duplicate entries.

Lines 387-388 instantiate mip/neos5.mps and mip/gen-ip054.mps, which appear both in active entries and as commented-out entries (lines 385-386). The commented-out duplicates should be removed to avoid confusion.

💡 Suggested cleanup

 INSTANTIATE_TEST_SUITE_P(DiversityTest,
                          DiversityTestParams,
                          testing::Values(
-                           // std::make_tuple("mip/gen-ip054.mps", 5.0f),
-                           // std::make_tuple("mip/pk1.mps", 5.0f),
                            std::make_tuple("mip/neos5.mps", 5.0f),
                            std::make_tuple("mip/gen-ip054.mps", 5.0f),
                            std::make_tuple("mip/pk1.mps", 5.0f),
-                           //  std::make_tuple("uccase9.mps"),
-                           // std::make_tuple("mip/neos5.mps", 5.0f),
                            std::make_tuple("mip/50v-10.mps", 5.0f)
-                           // std::make_tuple("mip/rmatr200-p5.mps", 5.0f)
                            ));

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tests/mip/diversity_test.cu` around lines 382 - 394, The
INSTANTIATE_TEST_SUITE_P block for DiversityTest (using DiversityTestParams)
contains commented-out entries that duplicate active tuples (e.g.,
"mip/neos5.mps" and "mip/gen-ip054.mps"); to clean up, remove the commented
duplicate lines so only one declaration remains per test case in the
INSTANTIATE_TEST_SUITE_P(DiversityTest, DiversityTestParams,
testing::Values(...)) invocation and leave the active std::make_tuple entries
(such as "mip/neos5.mps", "mip/gen-ip054.mps", "mip/pk1.mps", "mip/50v-10.mps")
intact.

---

283-314: Test structure is sound but consider removing debug logging in CI.

The test correctly validates recombiner determinism by comparing hashes across runs. However, the debug-level logging setup (lines 286-288) may produce excessive output in CI. Consider either:

• Removing the logger configuration, or
• Guarding it behind an environment variable check

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tests/mip/diversity_test.cu` around lines 283 - 314, The test
recombiners_deterministic currently forces debug logging by calling
cuopt::default_logger().set_pattern, set_level and flush_on at the start of the
test; remove these three logger configuration calls or wrap them in a runtime
guard (e.g., check an env var like CUOPT_DEBUG_LOG) so CI runs don't emit
excessive debug output; locate the calls in the TEST_P(DiversityTestParams,
recombiners_deterministic) body and either delete the
cuopt::default_logger().set_pattern(...),
cuopt::default_logger().set_level(...), and
cuopt::default_logger().flush_on(...) lines or condition them on
std::getenv("CUOPT_DEBUG_LOG") before applying them.

cpp/tests/mip/determinism_test.cu (1)

103-136: Consider removing unused helper function.

count_callbacks_with_origin (lines 115-121) is defined but not used anywhere in this file. If it's intended for future use, consider adding a comment; otherwise, it could be removed to avoid dead code.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tests/mip/determinism_test.cu` around lines 103 - 136, The helper
function count_callbacks_with_origin is currently unused in this file; either
remove its definition to eliminate dead code or, if you plan to use it later,
add a brief explanatory comment above count_callbacks_with_origin indicating its
intended future use (or mark it with a clear TODO), so reviewers know it’s
intentional; locate the function by name and delete it or add the comment
accordingly while leaving the other helpers (is_gpu_callback_origin,
count_gpu_callbacks, count_branch_and_bound_callbacks) unchanged.

cpp/src/mip_heuristics/relaxed_lp/relaxed_lp.cu (1)

82-84: Model produces unreliable estimates for constraint-heavy problems.

The negative coefficient for n_constraints (-400) causes the model to underestimate work for problems with many constraints relative to variables. For such problems, estim_ms clamps to 0, and the loop may overshoot the iteration estimate significantly.

Since this is marked as a temporary placeholder (per the TODO), consider adding a tracking issue or ensuring the predictor replacement is prioritized.

Do you want me to open a tracking issue for replacing this heuristic model?

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/src/mip_heuristics/relaxed_lp/relaxed_lp.cu` around lines 82 - 84, The
current heuristic for estim_ms uses a negative weight on
op_problem.n_constraints which causes underestimation and clamping to 0 for
constraint-heavy problems; update the formula in relaxed_lp.cu (the estim_ms
computation using op_problem.n_variables, op_problem.n_constraints,
op_problem.coefficients.size(), and estim_iters) to remove or make the
n_constraints coefficient non-negative (e.g., replace -400 with a small positive
or zero weight) and ensure a reasonable lower bound (e.g., min_ms > 0) instead
of hard clamping to 0; also add a TODO comment and open a tracking issue to
prioritize replacing this temporary predictor as referenced by the existing TODO
so the heuristic is not relied on long-term.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@cpp/src/mip_heuristics/relaxed_lp/relaxed_lp.cu`:
- Around line 79-90: The loop that increases estim_iters can overflow because
estim_iters is an int and may wrap if work_limit is huge; update the logic in
the block that computes estim_ms and increments estim_iters so it cannot run
unbounded: either change estim_iters to a wider integer type (e.g., int64_t)
and/or add an explicit upper bound check (compare against a safe MAX_ITERS
constant or std::numeric_limits<int>::max()/a guarded cap) and break when
reached, and ensure the estim_ms > work_limit*1000 check is preserved; locate
and modify the variables/loop referencing estim_iters, estim_ms, and work_limit
inside the do/while to add the guard and avoid signed overflow.

---

Nitpick comments:
In `@cpp/src/mip_heuristics/relaxed_lp/relaxed_lp.cu`:
- Around line 82-84: The current heuristic for estim_ms uses a negative weight
on op_problem.n_constraints which causes underestimation and clamping to 0 for
constraint-heavy problems; update the formula in relaxed_lp.cu (the estim_ms
computation using op_problem.n_variables, op_problem.n_constraints,
op_problem.coefficients.size(), and estim_iters) to remove or make the
n_constraints coefficient non-negative (e.g., replace -400 with a small positive
or zero weight) and ensure a reasonable lower bound (e.g., min_ms > 0) instead
of hard clamping to 0; also add a TODO comment and open a tracking issue to
prioritize replacing this temporary predictor as referenced by the existing TODO
so the heuristic is not relied on long-term.

In `@cpp/tests/mip/determinism_test.cu`:
- Around line 103-136: The helper function count_callbacks_with_origin is
currently unused in this file; either remove its definition to eliminate dead
code or, if you plan to use it later, add a brief explanatory comment above
count_callbacks_with_origin indicating its intended future use (or mark it with
a clear TODO), so reviewers know it’s intentional; locate the function by name
and delete it or add the comment accordingly while leaving the other helpers
(is_gpu_callback_origin, count_gpu_callbacks, count_branch_and_bound_callbacks)
unchanged.

In `@cpp/tests/mip/diversity_test.cu`:
- Line 69: The function setup_device_symbols(rmm::cuda_stream_view stream_view)
is a no-op; either remove this unused function (and clean up any call sites that
only exist to invoke it) or make its purpose explicit by adding a TODO/comment
indicating it's a deliberate placeholder for future device-symbol setup; update
references to setup_device_symbols in tests (e.g., any calls in
diversity_test.cu) to reflect the removal or to clarify that the call is
intentionally a no-op.
- Around line 71-73: The two test runner functions test_full_run_determinism and
test_initial_solution_determinism declare an unused seed parameter; either
remove the seed parameter from both function signatures (and update any callers)
if external seeding is intended, or make the functions actually use the value by
calling cuopt::seed_generator::set_seed(seed) (or equivalent local seeding) at
the start of each function to apply the provided seed; pick one approach and
apply it consistently to both functions.
- Around line 382-394: The INSTANTIATE_TEST_SUITE_P block for DiversityTest
(using DiversityTestParams) contains commented-out entries that duplicate active
tuples (e.g., "mip/neos5.mps" and "mip/gen-ip054.mps"); to clean up, remove the
commented duplicate lines so only one declaration remains per test case in the
INSTANTIATE_TEST_SUITE_P(DiversityTest, DiversityTestParams,
testing::Values(...)) invocation and leave the active std::make_tuple entries
(such as "mip/neos5.mps", "mip/gen-ip054.mps", "mip/pk1.mps", "mip/50v-10.mps")
intact.
- Around line 283-314: The test recombiners_deterministic currently forces debug
logging by calling cuopt::default_logger().set_pattern, set_level and flush_on
at the start of the test; remove these three logger configuration calls or wrap
them in a runtime guard (e.g., check an env var like CUOPT_DEBUG_LOG) so CI runs
don't emit excessive debug output; locate the calls in the
TEST_P(DiversityTestParams, recombiners_deterministic) body and either delete
the cuopt::default_logger().set_pattern(...),
cuopt::default_logger().set_level(...), and
cuopt::default_logger().flush_on(...) lines or condition them on
std::getenv("CUOPT_DEBUG_LOG") before applying them.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6d186eaf-cff0-4117-9f13-2cb2e79896ba

📥 Commits

Reviewing files that changed from the base of the PR and between 40a72db and 079a84e.

📒 Files selected for processing (5)

• cpp/src/branch_and_bound/pseudo_costs.cpp
• cpp/src/mip_heuristics/local_search/rounding/constraint_prop.cu
• cpp/src/mip_heuristics/relaxed_lp/relaxed_lp.cu
• cpp/tests/mip/determinism_test.cu
• cpp/tests/mip/diversity_test.cu

🚧 Files skipped from review as they are similar to previous changes (2)

• cpp/src/branch_and_bound/pseudo_costs.cpp
• cpp/src/mip_heuristics/local_search/rounding/constraint_prop.cu

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Potential unbounded loop and signed integer overflow.

When work_limit is finite but very large (e.g., work_limit = 1e6 seconds), this loop may iterate until estim_iters overflows. Since estim_iters is a signed int, overflow is undefined behavior.

Consider adding bounds:

🛡️ Proposed fix to bound iterations

+    constexpr int max_estim_iters = std::numeric_limits<int>::max() - 100;
     if (!std::isinf(work_limit)) {
       do {
         // TODO: use an actual predictor model here
         double estim_ms = 313 + 200 * op_problem.n_variables - 400 * op_problem.n_constraints +
                           600 * op_problem.coefficients.size() + 7100 * estim_iters;
         estim_ms = std::max(0.0, estim_ms);
         if (estim_ms > work_limit * 1000) { break; }
+        if (estim_iters >= max_estim_iters) { break; }
         estim_iters += 100;
       } while (true);

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/src/mip_heuristics/relaxed_lp/relaxed_lp.cu` around lines 79 - 90, The
loop that increases estim_iters can overflow because estim_iters is an int and
may wrap if work_limit is huge; update the logic in the block that computes
estim_ms and increments estim_iters so it cannot run unbounded: either change
estim_iters to a wider integer type (e.g., int64_t) and/or add an explicit upper
bound check (compare against a safe MAX_ITERS constant or
std::numeric_limits<int>::max()/a guarded cap) and break when reached, and
ensure the estim_ms > work_limit*1000 check is preserved; locate and modify the
variables/loop referencing estim_iters, estim_ms, and work_limit inside the
do/while to add the guard and avoid signed overflow.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

[github-actions]

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.