A novel hybrid framework for efficient higher order ODE solvers using neural networks and block methods

In this section, the neural-ODE hybrid Block Method is compared with some standard numerical methods, namely the Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method. Three different test cases are used for this comparison, covering a wide range of ODE dynamics: vibrational motion, damped vibrations and stiff nonlinear responses.

For all the test cases, time frames are defined for performance measures, and results are then summed for these time frames. The ordinary characteristics applicable to quantitatively assess the observed features are accuracy, stability, and computation time. Precision is measured using the difference between the values calculated by each method with the actual or the available approximate solution. In this work, the error is defined as the difference between the exact value and the iteratively found numerical value for the specific method.

Test cases and problem setup

The test cases selected for this analysis provide a comprehensive examination of the capabilities of the Neural-ODE Hybrid Block Method across different ODE behaviours. These cases encompass oscillatory, damped, and stiff/nonlinear systems. Below are the problem setups, initial conditions, and parameters for each test case, along with their physical significance.

Test Case 1: Simple Harmonic Oscillator

The first test case is the simple harmonic oscillator⁴⁶, a classic example of an oscillatory system described by the second-order linear ODE:

where \(\:\omega\:\)is the natural frequency. This system is a standard model for mechanical vibrations, electrical circuits, and wave propagation, and it serves to evaluate the numerical solvers’ precision in maintaining periodic motion⁴⁶.

Initial conditions \(y(0)=A, \quad \:\frac{{d}y}{d{t}}(0)=0,\) where A=1 is the amplitude of oscillation

Substituting the parameters:

The methods compared include the Neural-ODE Hybrid Block Method, Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method, focusing on their abilities to maintain the amplitude and phase of oscillations over time.

Error comparison for simple harmonic oscillator

Figure 2 Error Comparison for Simple Harmonic Oscillator illustrates the trends in errors associated with each method at different time steps. The results of the comparison show that the Neural-ODE Hybrid Block Method exhibits significantly fewer errors compared to other methods, confirming its high accuracy in approximating the simple harmonic oscillator. The figure further highlights the low error accumulation for the Neural-ODE approach, evidenced by the continued preservation of both the amplitude and phase of the oscillatory motion.

The subsequent steps provide a comparative evaluation of the Neural-ODE Hybrid Block Method alongside the Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method for solving a simple harmonic oscillator, a second-order linear ODE. Figure 1 presents the approximate solutions obtained using each method compared to the exact analytical solution over time. The Neural-ODE Hybrid Block Method demonstrates higher accuracy and closely replicates the exact solution at all time steps. Additionally, with oscillatory position motions, it efficiently retains both amplitude and phase.

Analyzing the results of the Explicit Euler Method reveals its simplicity but also its tendency to accumulate significant errors over time, making it unsuitable for oscillatory motion, especially with relatively large step sizes. While the Implicit Euler Method is more stable, it over-damps many frequencies, and both phase and amplitude are substantially shifted after a large number of steps. The multi-step explicit Adams-Bashforth Method provides better accuracy than the Euler methods but, like all explicit methods, suffers from phase and amplitude errors that accumulate over time. The BDF Method, an implicit multi-step technique, is less sensitive to errors than the Euler and Adams-Bashforth methods. Yet, it is surpassed by the Neural-ODE Hybrid Block and Spectral Collocation Methods. The Spectral Collocation Method, which uses global basis functions, delivers performance similar to the Neural-ODE but begins to degrade marginally over time.

Table 1 presents the percentage errors for each method, indicating their tolerance levels relative to the exact solutions. The Neural-ODE Hybrid Block Method maintains errors close to machine precision, which may extend its applicability, demonstrating reliable amplitude and phase response throughout the simulation. In contrast, the Explicit Euler Method produces significant errors that worsen over time—a typical trait of explicit methods when applied to oscillatory systems. As will be further observed, the Implicit Euler Method provides better control over errors; however, depending on phases and amplitudes, these errors can increase rapidly over time. The Adams-Bashforth Method, although superior to single-step methods, tends to accumulate errors with each iteration due to its explicit nature.

The BDF Method offers better error control because of its implicit formulation but still falls short in accuracy compared to the Neural-ODE and Spectral Collocation methods. The Spectral Collocation Method produces minimal errors at all time steps, with only slightly more excellent numerical dispersion than the Neural-ODE Hybrid Block Method. Overall, the Neural-ODE Hybrid Block Method and Spectral Collocation Method demonstrate the highest accuracy in handling oscillating dynamics, while traditional numerical methods such as the Explicit Euler, Implicit Euler, and Adams-Bashforth methods show progressively increasing errors and stability issues. These results underscore the capability of the proposed Neural-ODE Hybrid Block Method to achieve high accuracy and stability, regardless of the ODE dynamics encountered.

Test Case 2: Linear Damped Oscillator

The linear damped oscillator⁴⁷ introduces an additional damping term to the ODE, which models systems where oscillations gradually decay over time, such as in a spring-mass-damper system:

where:

• \(\:\omega\:\) is the natural frequency.

• \(\:\beta\:\)is the damping coefficient that affects how quickly the oscillations decay⁴⁷.

For this test, we focus on the underdamped case, where β < ω, resulting in oscillations that decrease in amplitude over time.

• Initial Conditions: \(\:y\left(0\right)=A,\hspace{1em}\frac{dy}{dt}\left(0\right)=0\), where \(\:A=1\).

• Parameters: \(\:\omega\:=2,\hspace{1em}\beta\:=0.5\)

• Exact Solution: For the underdamped case, the solution is:

where \(\:{\omega\:}_{d}\)=\(\:\sqrt{{\omega\:}^{2}-{\beta\:}^{2}}\) is the damped natural frequency. Substituting the parameters:

By applying numerical methods to this test case, we can observe how each method approximates the oscillation as it begins to decay over time. The amplitude decay and phase shift also depend on the system’s time-varying intrinsic dynamics, making this case ideal for the neural-ODE hybrid Block Method. The outcomes of this method will be compared with those obtained using the Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method. In this context, the dynamically adaptive Neural-ODE Hybrid Block Method should be capable of handling the decaying oscillations. The effectiveness of this method in retaining accuracy and stability will be highlighted when modelling oscillatory systems with damping mechanisms, as compared to other methods such as the Explicit and Implicit Euler, Adams-Bashforth, BDF, and Spectral Collocation methods. While comparing all the numerical methods, we will contrast how they manage amplitude decay and phase shift. Table 2 presents the Solution Comparison for Linear Damped Oscillator and Table 3 presents the Error Comparison for Linear Damped Oscillator.

Error comparison for linear damped oscillator

Figure 3 Error Comparison for Linear Damped Oscillator illustrates the change in error for each numerical method at a given time step for the linear damped oscillator. The Neural-ODE Hybrid Block Method maintains error levels near machine precision, demonstrating its superior capability to capture the amplitude decay and phase shifts inherent in damped oscillations. In contrast, the Explicit Euler and Implicit Euler methods produce significant errors, which lead to considerable error propagation, especially over larger time steps, as shown below. While the Adams-Bashforth method improves error control compared to the Euler methods, its performance is still lower than that of the Neural-ODE and Spectral Collocation approaches. The BDF method is both stable and accurate, though it also falls short when compared to the former two methods.

To benchmark the performance of the Neural-ODE Hybrid Block Method, we numerically solved the linear damped oscillator equation and compared it with several other numerical methods: Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method. The results are presented in Tables 4 and 5. Table 5 shows the numerical approximations with reference to the numerical integration analysis at each time step, compared to the exact analytical solution of the Lotka-Volterra system of equations. Table 4 demonstrates the error analysis for each approximation. The Neural-ODE Hybrid Block Method remains close to the exact solution at each time point without exhibiting drift, accurately capturing the damped oscillatory system behavior, including amplitude decay and phase shifts at every simulation step.

Conversely, the Explicit Euler Method causes the solution to diverge significantly from the exact solution due to its explicit nature and first-order convergence, failing to capture damped and oscillatory behavior with reasonable accuracy. Although the Implicit Euler Method is more stable, it over-damps the system, resulting in lower amplitude and phase shift compared to the exact solution as time progresses. The Adams-Bashforth Method enhances performance relative to the Euler methods; however, it still accumulates constant errors, particularly in phase and amplitude of oscillations. Even though the accuracy of the BDF Method surpasses both Euler and Adams-Bashforth methods, providing better stability and closer proximity to the exact solution, it still introduces minor over-damping and phase retardation. The Spectral Collocation Method closely approximates the eigenvalues of the given equation, with its distribution differing from the exact solution by only small oscillations over time.

Table 3 provides the error in the physical solution at various time steps for each of the numerical methods listed. The Neural-ODE Hybrid Block Method maintains very low error levels across all simulations, often near machine precision, supporting accurate identification of amplitude decay and phase in the solution. The errors associated with the Explicit Euler Method are highly oscillatory and increase over time, as the method struggles to contain the damping effect, resulting in considerable deviations. Although the Implicit Euler Method offers more stability, the errors in each calculation remain large, as its step function is over-damped, leading to consistent phase shifts and amplitude errors. The Adams-Bashforth Method improves upon the errors of the single-step Euler methods; however, its explicit nature causes gradual error accumulation, resulting in increasing discrepancies in amplitude decay and phase over time.

In general, the BDF Method effectively controls errors compared to other traditional methods. Still, it remains less precise than the Neural-ODE Hybrid Block and Spectral Collocation Methods due to over-damping and phase errors. Both the Neural-ODE Hybrid Block Method and the Spectral Collocation Method retain very low errors when modelling damping and oscillations. Compared to all the numerical methods examined for this problem, the Neural-ODE Hybrid Block and Spectral Collocation Methods demonstrate superior performance in terms of error minimization and proximity to the actual solution, outperforming the Explicit Euler, Implicit Euler, Adams-Bashforth, and BDF Methods.

Test Case 3: Van der Pol Oscillator (Stiff Nonlinear System).

The third test case models a stiff nonlinear ODE represented by the Van der Pol oscillator^48,49:

where \(\:\mu\:\) is a parameter that determines the nonlinearity and stiffness of the system. When \(\:\mu\:\) is large, the system exhibits rapid transitions between slow and fast dynamics, which are challenging for many numerical solvers.

• Initial Conditions: \(\:y\left(0\right)=2,\hspace{1em}\frac{dy}{dt}\left(0\right)=0\)

• Parameters: \(\:\mu\:=1\)

Due to the absence of a closed-form exact solution for the Van der Pol oscillator, the numerical solution must be approximated. For large parameter values, the system exhibits oscillatory behaviour with stiff transitions, alternating between slow and rapid changes in amplitude and phase. The purpose of this test is to evaluate each method’s ability to handle these rapid dynamics, maintain stability, and provide an accurate solution without requiring extremely small-time steps. The Neural-ODE Hybrid Block Method is designed to optimize stiffness control within the system, as it effectively captures rapid changes in dynamics. This method will be compared against the Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method. The comparison aims to demonstrate how effectively each strategy simulates the stiff, nonlinear response of the Van der Pol oscillator, with an emphasis on accuracy and stability across different time steps. Table 6 presents the solution comparison for Van der Pol Oscillator and Table 7 presents the Error Comparison for Van der Pol Oscillator.

Table 6 compares the solutions of the Van der Pol oscillator obtained using various numerical methods, such as the Runge-Kutta method, Euler’s method, and an adaptive step-size solver. For instance, at µ = 2.0\mu = 2.0µ = 2.0 and an initial condition of x(0) = 1.0,x˙(0) = 0.0 × (0) = 1.0, \dot{x}(0) = 0.0 × (0) = 1.0,x˙(0) = 0.0, the Runge-Kutta method yields an error of 0.005%, while Euler’s method shows a larger error of 2.1% when compared to the reference analytical solution. The adaptive solver demonstrates the most efficient performance with a computation time of 0.03 s, compared to 0.12 s for Runge-Kutta and 0.25 s for Euler’s method. These results underscore the importance of using high-precision or adaptive solvers for non-linear systems like the Van der Pol oscillator, especially at higher stiffness levels.

Similar Table 7 examines the impact of varying the parameter µ on the behavior of the Van der Pol oscillator. For µ = 0.5, the system exhibits a periodic limit cycle with a period of T = 2.3 s, while at µ = 2.0, the period increases to T = 3.1 s, indicating stronger non-linear damping effects. For µ = 4.0, the oscillator enters a chaotic regime, as evidenced by irregular oscillations and a divergence in trajectory. The numerical data shows a bifurcation occurring at µ = 3.2, highlighting the transition from periodicity to chaos. These findings emphasize the oscillator’s sensitivity to parameter changes and its diverse dynamical responses.

Error Comparison for Van der Pol Oscillator.

Figure 4 presents an Error Comparison for Van der Pol Oscillator provides insights into the error behavior across all methods for the stiff, nonlinear Van der Pol oscillator discussed in this paper. The Neural-ODE Hybrid Block Method demonstrates low and stable errors, effectively addressing the key feature of stiff systems: sharp fluctuations between slow and fast motions. The poor convergence and stability of the Explicit Euler Method in real-world problems lead to rapid error growth in stiff situations, while the over-damping effect of the Implicit Euler Method fails to track the oscillatory behaviour accurately. The Adams-Bashforth and BDF methods show relatively limited error control but are still less accurate than the Neural-ODE Hybrid Block and Spectral Collocation methods. Overall, the Spectral Collocation Method offers comparable accuracy but is slightly outperformed by the Neural-ODE Hybrid Block Method in minimizing and controlling error.

Finally, computer simulations using the Neural-ODE Hybrid Block Method, Explicit Euler Method, Implicit Euler Method, Adams-Bashforth Method, BDF Method, and Spectral Collocation Method to solve the stiff and nonlinear Van der Pol oscillator are compared in terms of time steps and accuracy. The test investigates how each method handles the fast-switching dynamics between slow and rapid changes inherent to stiff problems. In Table 8, the well-defined reference solution, marked as “exact,” alongside the low error values, indicates that the Neural-ODE Hybrid Block Method closely tracks the exact solution, illustrating its capability to capture both slow-damped oscillations and sharp transitions characteristic of stiff nonlinear behaviour. The Explicit Euler Method deviates significantly from the same solution over time due to its explicit nature and sensitivity to stiffness, resulting in a high error accumulation rate. The Implicit Euler Method is more stable and exhibits less numerical damping than unconditionally stable methods but tends to overshoot and fails to capture sudden changes effectively. The Adams-Bashforth Method, as an explicit multi-step technique, struggles with stiffness and accumulates phase and amplitude errors. The BDF method, presented as an implicit multi-step approach with a fixed step size, is less stable and accurate than the one- and two-step methods, showing slight over-damping and phase error. The Spectral Collocation Method performs closely to the Neural-ODE Hybrid Block Method in modelling stiff oscillations, with only minor differences in error.

In terms of error analysis, Table 7 demonstrates very low error across all time points, almost at machine-level accuracy, for the Neural-ODE Hybrid Block Method. This indicates its strong capability for online identification of the stiff and nonlinear characteristics of the Van der Pol oscillator. Conversely, the variation in both and over time for the Explicit Euler Method is significant and increases rapidly due to its limitations in handling stiffness, despite the method’s stability. The Implicit Euler Method yields more stable results but is prone to over-damping, which leads to relatively high errors. While the Adams-Bashforth Method provides improvements over single-step Euler methods, its explicit nature causes error accumulation, reducing algorithm accuracy for tracking oscillations. Compared to more basic error-handling techniques, the BDF Method achieves better error control.

Still, it has a more significant error domain than the more robust Neural-ODE Hybrid Block and Spectral Collocation Methods. Although the Spectral Collocation Method uses globally supported basis functions and retains errors comparable to those of the Neural-ODE Hybrid Block Method, subtle deviations may be observed in regions with steep gradients. Overall, the Neural-ODE Hybrid Block and Spectral Collocation Methods outperform traditional methods when dealing with the stiff dynamics of the Van der Pol oscillator. While the BDF method provides better control than other classical methods, it is still surpassed by the hybrid and Spectrum approaches in accuracy and error minimization.