Sunday, February 19, 2012
In our celestial navigation classes, I always stress that the inherent flaw of utilizing spherical trigonometry on a body which is not actually a sphere introduces navigation errors many times larger than those introduced by imperfections in our instruments, or rounding errors in our logarithmic tables. This happens to be true, but I had never bothered to actually quantify this until yesterday. One of my coworkers recently sat for his 500 ton Master's license, and one of the questions he was asked was the great-circle circumference of the earth. Presumably the answer the USCG was looking for was 21,600 nautical miles, which is just 360° x 60' of arc. Which would be true, if the earth happened to be a sphere. Which, of course, it isn't.
Along any of earth's meridians, the circumference is about 21,603 nm, which is pretty close to the abstract spherical circumference. Along the equator, however, the earth's circumference is about 21,639 nm. 39 nautical miles is about 45 statute miles, or 72 kilometers. A navigation error of this magnitude is far from trivial.
Fortunately, by the nature of the way celestial navigation is performed, the greatest distance we ever have to worry about is 1/4 of the circumference of the earth, and this in turn limits our possible position error to 1/4 of that distance.
So, here's the very worst case imaginable. You are somewhere on the equator, and you take sights of the stars Mintaka and Polaris at the moment that both of these objects are just touching the horizon. Let's ignore all of the other reasons why this is a really bad idea, and also ignore the very large error in atmospheric refraction which would also occur in this case. We see that the actual geographic position of Polaris is only six tenths of a mile further away than our standard sight reduction would tell us, but Mintaka is nearly ten miles further away. Yes, ten. And even if we shoot Mintaka at the more reasonable altitude of 45°, the error is still in the neighborhood of 5 nautical miles.
By extension, if we are in the tropics, all celestial bodies on the equinoctial will be in error by as much as ten nautical miles, and if we are in temperate latitudes they may still be in error by as much as five nautical miles. From the standpoint of ocean navigation this isn't horrible. But it does give some perspective on the relative importance of precision in celestial navigation, as opposed to accuracy.