Accuracy and Validation

From BurnSim Wiki
Jump to navigation Jump to search

BurnSim is used to design motors that people then build and fire, so "is this number right?" is a fair question to ask of it. This page sets out what BurnSim's results are checked against, how closely they agree, and - just as important - which numbers it cannot check for you.

Short version: the thermochemistry is validated against NASA's own published reference cases and agrees with them closely. The ballistic simulation is a steady-state model, and its accuracy depends on propellant data you supply. Nothing here makes a predicted pressure curve a guarantee: treat all results as estimates, test in a safe place, and assume the motor can fail.

Where the thermochemistry comes from

When BurnSim computes C*, specific heat ratio and molar mass from an ingredient recipe (see Propellant Thermochemistry), it is not using a correlation or a lookup table. It solves the chemical equilibrium of the combustion products by minimising Gibbs free energy, the same method NASA's CEA (Chemical Equilibrium with Applications) uses, and it uses NASA's own thermodynamic species data - the TP-2002-211556 library - for the properties of those products.

The solver and the rocket-problem driver are transliterated from CEA's own source, which NASA released under the Apache 2.0 licence, and the algorithm is documented in full in NASA RP-1311. This matters for the next section: because BurnSim implements the same algorithm over the same data, NASA's published example cases are a test BurnSim can be held to exactly, rather than merely compared against.

The validation gate: NASA RP-1311 reference cases

RP-1311 ships a set of worked example problems together with CEA's own reference output for each one. BurnSim reproduces seven of them, chosen to exercise different parts of the calculation rather than the same path seven times. Each is an automated test that runs on every build, and the expected values are transcribed from NASA's reference output - they are not produced by BurnSim and then blessed.

Case What it is Why this one
Example 1 Hydrogen burned in air at an equivalence ratio of 1 The plainest possible equilibrium solve - if this disagrees, nothing else is worth looking at.
Example 3 Toluene and n-octane burned in air preheated to 700 K A lean, nitrogen-dominated flame at around 2400 K whose interesting products are the nitrogen oxides. No rocket propellant looks like this, which is the point: a solver accidentally tuned to propellant chemistry would fail here.
Example 5 Adiabatic combustion of a real solid propellant The case closest to what BurnSim is actually for.
Example 8 Liquid hydrogen and liquid oxygen expanded through a nozzle Validates the rocket driver rather than the equilibrium solve. The throat is found by a search, the area ratios by a second search on top of it, and C* ties them together - a sign error anywhere in that still produces a table of plausible-looking numbers.
Example 12 Monomethylhydrazine and nitrogen tetroxide A storable bipropellant, well away from the chemistry of the other cases.
Example 13 An expansion that stalls on a melting point A product that changes phase partway down the nozzle, which is a case naive implementations get wrong silently.
Example 14 Combustion with condensed products The one that matters most for amateur and experimental work: metallised propellants produce condensed aluminium oxide, and the condensed fraction changes flame temperature and gas properties considerably.

How close is "agrees"?

Close enough that the tolerances are worth quoting. On example 8, BurnSim is required to match NASA's reference output to within:

  • 0.5 K on a chamber flame temperature of 3383.8 K - about 0.015%
  • 1 m/s on a characteristic velocity (C*) of 2332.3 m/s - about 0.04%
  • 0.0001 on the isentropic exponent
  • 0.01 bar on a throat pressure of 30.66 bar

If any of those drift, the build fails. Agreement at this level is what distinguishes an implementation of the algorithm from an approximation of it.

How BurnSim compares to ProPEP 3

Many people arrive at BurnSim already using ProPep3, so the practical question is how the two compare on the same formulation.

Expect close agreement, not identical numbers, and a percent-level difference in C* is normal. The two use different thermodynamic data of different vintages: ProPEP's was last updated in 1986, while BurnSim uses NASA's TP-2002-211556 library. Neither is "wrong" because of the other, and a small difference is not a bug in either one.

What would be a real disagreement, and worth reporting, is a qualitative one: a flame temperature off by hundreds of degrees, the wrong product species, or aluminium oxide missing entirely from a metallised mix. Those indicate an error rather than a data vintage.

One difference trips people up often enough to repeat here: ProPEP's Isp* and an expanded Isp are not the same quantity, and comparing one against the other makes the two programs look badly out of agreement when they are not - roughly 115 s against 153 s for KNSB at 1000 psi. Propellant Thermochemistry explains which is which.

The ingredient table

The recipe you enter is only as good as the ingredient data behind it, so every value in BurnSim's built-in ingredient table has been checked against a published second source, and each entry records where its numbers came from. You can audit any of them rather than taking them on trust.

That checking caught real problems rather than merely confirming what was already there: several substances exist in three or four solid forms whose heats of formation are tens of kJ/mol apart, which is the kind of error that produces a confidently wrong flame temperature. Ingredients are picked from a list rather than typed, so a misspelled name cannot silently produce a wrong analysis.

If you add your own ingredients, or import them from ProPEP's PEPCODED.DAF (see Importing ingredients from ProPEP), those values are yours and are not second-sourced by anyone - the audit trail above covers the built-in table.

What BurnSim cannot check for you

This is the part to read carefully.

  • Burn rate (a and n) requires real test data. No thermochemical calculation predicts how fast a propellant burns. BurnSim computes C*, specific heat ratio and molar mass from a recipe; a and n come from firing motors and measuring, which is what Propellant Characterization is about. A simulation built on guessed burn-rate coefficients is a guess regardless of how good the thermochemistry is.
  • Density is yours too. It comes from the formulation's theoretical or measured value, not from the analysis.
  • The ballistic simulation assumes steady-state operation. It does not model ignition transients or pressure-drop (burnout, boost-sustain) behaviour.
  • Erosive burning is modelled but off by default. In motors with high length-to-diameter ratios, crossflow near the nozzle end can raise burn rate and chamber pressure substantially above a steady-state prediction. BurnSim 4 can model this (Erosive Burning, with the Lenoir-Robillard, Mukunda-Paul or Ma (2020) correlations) but it is not enabled unless you enable it. These are empirical correlations, and they disagree with each other - which is itself informative, and a reason to run more than one.
  • Your own measurements outrank all of it. If you have characterised a propellant against real test data and BurnSim's computed numbers disagree with yours, prefer yours. BurnSim asks before overwriting a value you entered by hand for exactly this reason.

Reproducing this yourself

The validation is not a claim you have to take on faith. The reference cases are NASA's, published in RP-1311 and shipped with CEA, and the expected values are in CEA's own reference_output files. If you run the same case through CEA and through BurnSim, you should get the same answer to the tolerances above - and if you do not, that is a bug report we want.

See also

Retrieved from "https://wiki.burnsim.com/index.php?title=Accuracy_and_Validation&oldid=294"