Analyzing the accuracy of JSBSim for the computation of a vehicle rotations.

[bcoconni]

This PR submits a Python notebook that analyzes the accuracy of JSBSim's computations for the rotations of a vehicle.

Analysis summary and conclusions

The analysis compares JSBSim to the exact computation of a free rigid body rotation from the paper Numerical Implementation of the Exact Dynamics of Free Rigid Bodies, van Zon R., Schofiled J., Dec 15, 2006

Basically it shows that the current defaults for the integration of rotations are poor choices:

jsbsim/src/models/FGPropagate.cpp

Lines 131 to 134 in 3896d4a

	integrator_rotational_rate = eRectEuler;
	integrator_translational_rate = eAdamsBashforth2;
	integrator_rotational_position = eRectEuler;
	integrator_translational_position = eAdamsBashforth3;

• The Euler method that is the default is unsurprisingly performing poorly for both rotation rates and attitude (quaternion and Euler angles)
• The trapezoidal method, from an unknown pedigree, is actually worse than Euler's method. The notebook of this PR demonstrates with mathematical rigor that the trapezoidal method:
  • Is of first order, despite being a multistep method
  • Has a consistency error twice as large as Euler

To put it bluntly, the trapezoidal method is worse than Euler in every possible scenario and should not be used for any purpose.

The other integration schemes (Adams-Bashforth, Buss' methods and local linearization) show adequate precision for the computation of rotations with methods of higher order performing better than methods of lower order.

Notably, methods specifically designed for the integration of quaternions such as Buss of the 1st and "augmented" 2nd order as well as NASA's local linearization are behaving better than Adams-Bashforth of the 2nd order. The Buss method of the 2nd order gives similar accuracy as Adams-Bashforth of the 4th and 5th order.

Local linearization and the Buss methods are available in JSBSim since commit 9d2416b (Dec 2011)

Pull Request content

The notebook comes with the C program from van Zon & Schofield that computes the exact solution. It needs the GNU scientific library to compile.

The CMake file to compile the C program is also included as well as the JSBSim model for the top which, I think, can claim the crown for the smallest JSBSim model ever 😄

All the files are located under the check_cases folder since, in my mind, this notebook is more a tool to check JSBSim performance than an example.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 25.11%. Comparing base (d371f38) to head (5719564).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1453   +/-   ##
=======================================
  Coverage   25.11%   25.11%           
=======================================
  Files         171      171           
  Lines       18840    18842    +2     
=======================================
+ Hits         4731     4733    +2     
  Misses      14109    14109           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bcoconni]

I am planning to submit another PR with a proposal for new defaults for the methods of integration of rotation rates and quaternions. But I would like to hear your feedback about my analysis before proceeding.

[agodemar]

@bcoconni this is a great example! I love additional insights like this. OK from me to merge

[seanmcleod70]

Looks like a very useful exercise, and a fair bit of work 😉 I see the A4's legendary roll rate even gets a mention.

So, in terms of the NASA test cases evaluation of a number of flight dynamics engines including JSBSim, were all the results close enough to each other because they all ended up using the same integration schemes, or were there different integration schemes used but the angular rates were small enough and the simulation run short enough that there wasn't a significant difference in the results?

[bcoconni]

So, in terms of the NASA test cases evaluation of a number of flight dynamics engines including JSBSim, were all the results close enough to each other because they all ended up using the same integration schemes, or were there different integration schemes used but the angular rates were small enough and the simulation run short enough that there wasn't a significant difference in the results?

I have added the script and model for the NASA atmospheric case 2 (tumbling brick) to the PR. The XML files are located under the folder check_cases/NASA_EOM_2015 and below is a comparison of the pitch, roll and yaw angles between JSBSim (with standard settings) and the NASA simulation 1 (i.e. Atmos_02_sim_01.csv from the NASA web site):

And the comparison between NASA simulation 1 and JSBSim using Adams-Bashforth 4 for the rotation rates and Buss' method of the 1st order for the rotations:

One can see that the errors magnitudes are reduced by an order of magnitude (10x or 20x) using adequate integration schemes as per the explanations given in the notebook.

[agodemar]

Great contribution @bcoconni - The PR looks good to me

[bcoconni]

Thanks @agodemar.

More details about the NASA test case 2: it appears that the simulation 2 (SIM 2) is deviating from the others by more than 4 degrees over the simulated 30 seconds:

With the default settings of the integration methods, we are deviating by 2.5 degrees over 30 seconds. So JSBSim 1.3.1 with its default parameters is comparing reasonably well (or at least not any worse than SIM 2) with respect to the other simulations of the paper from 2015. And when selecting better integration methods, JSBSim 1.3.1 is reasonably close to SIM 1.

Also note that when the Adams-Bashforth 4 and the Buss 1 methods are selected, there is no longer a drift between JSBSim and SIM1: the gap between the 2 simulations is bounded and do not increase over time.

[jonsberndt]

My recollection of the comparison of the atmospheric test cases was that the entire set I performed with JSBSim (8 or 10 cases ... ?) were all right on top of the other NASA sim.

[agodemar]

As soon as this PR is merged I'll make a LinkedIn post. This gives even more strength to the library.

[seanmcleod70]

So it looks like there were 7 different simulation tools. But not all 7 ran every single check case. For example for test case 2 (Tumbling brick, no damping) there are only results for 5 simulations, sims [1, 2, 4, 5, 6].

Haven't found any reference so far in the 2 NASA documents providing a mapping from sim_XX.csv to the simulation tool. Not sure if that's on purpose, or I've just missed it somewhere?

The assembled tool-set included:

• Core from ARFC
• JEOD from JSC
• JSBSim [ref. 6]
• LaSRS++ from LaRC
• MAVERIC from MSFC
• POST-II from LaRC
• VMSRTE from ARC

A subset of the simulation tools have details on the integrators used, e.g.

VMSRTE was employed for some of the atmospheric check-cases; it used an Adams-Bashforth second-order integration method [23] with a 0.01-sec time step.

And noticed the following for JSBSim.

JSBSim was exercised for the atmospheric check-cases; it used a “higher-order” Adams-Bashforth integration method at 120 frames per simulated second.

So, it looks like that at the time for the NASA test cases JSBSim wasn't using the current default of Euler for rotational integration?

I haven't checked the git history yet, was Adams-Bashforth the default at some stage, or was it just specifically set as the default for the NASA tests?

[seanmcleod70]

Ah, just came across this.

The identities of the various tools were masked to encourage participation in this voluntary effort. Also, the atmospheric cases used numerical designations (SIM 1, SIM 2, etc.) while the orbital cases used alpha...

[seanmcleod70]

Comparison between SIM2 and JSBSim today using Euler.