The shortest route between two points on the Earth's surface is called a \(great\: circle\). I'm going to derive the formula connecting the latitude and longitude of points lying on the shortest path connecting two points - first using vectors, then using calculus.
First, we can convert the latitude and longitude into rectangular coordinates. I will use \(\phi\) to represent longitude and \(\theta \) for latitude, as shown in the diagram. We can see that \(x = r\cos \theta \cos \phi\) , \(y = r \cos \theta \sin \phi\), and \(z = r\sin \theta\). |
When flying from A to B, the shortest path will be in the OAB plane. We can find the equation of this plane by taking the cross product of OA and OB.
\[\overrightarrow{OA} \times \overrightarrow {OB} =\begin{pmatrix}r\cos \theta_A \cos \phi_A\\r \cos \theta_A\sin\phi_A \\ r\sin\theta_A\end{pmatrix} \times \begin{pmatrix} r\cos \theta_B \cos \phi_B \\ r \cos \theta_B \sin \phi_B \\ r\sin \theta_B \end{pmatrix}\]
\[ = r^2 \begin{pmatrix}\cos\theta_A \sin \phi_A \sin\theta_B - \cos\theta_B\sin \phi_B\sin\theta_A \\ \cos\theta_B\cos\phi_B\sin\theta_A - \cos\theta_A\cos\phi_A\sin\theta_B \\ \cos\theta_A\cos\phi_A\cos\theta_B\sin\phi_B - \cos\theta_B\cos\phi_B\cos\theta_A\sin\phi_A \end{pmatrix}\]
Since we are only interested in the direction of this vector and not the magnitude, we will ignore the \(r^2\)
We can then use the formula for a plane \(\mathbf{r.n}=\mathbf{a.n}\). Since this plane will go through the origin, we have \(\mathbf{r.n}=0\) and we can replace \(\mathbf{r}\) by \(\begin{pmatrix}r\cos \theta \cos \phi\\r \cos \theta\sin\phi \\ r\sin\theta\end{pmatrix}\) to get (after cancelling by \(r\))
\[(\cos \theta_A \sin \phi_A \sin\theta_B - \cos \theta_B\sin \phi_B\sin\theta_A) \cos \theta \cos \phi + (\cos\theta_B\cos\phi_B\sin\theta_A - \cos\theta_A\cos\phi_A\sin\theta_B ) \cos \theta\sin\phi + (\cos\theta_A\cos\phi_A\cos\theta_B\sin\phi_B - \cos\theta_B\cos\phi_B\cos\theta_A\sin\phi_A) \sin\theta \]
This can be simplified a little to get:
\[\overrightarrow{OA} \times \overrightarrow {OB} =\begin{pmatrix}r\cos \theta_A \cos \phi_A\\r \cos \theta_A\sin\phi_A \\ r\sin\theta_A\end{pmatrix} \times \begin{pmatrix} r\cos \theta_B \cos \phi_B \\ r \cos \theta_B \sin \phi_B \\ r\sin \theta_B \end{pmatrix}\]
\[ = r^2 \begin{pmatrix}\cos\theta_A \sin \phi_A \sin\theta_B - \cos\theta_B\sin \phi_B\sin\theta_A \\ \cos\theta_B\cos\phi_B\sin\theta_A - \cos\theta_A\cos\phi_A\sin\theta_B \\ \cos\theta_A\cos\phi_A\cos\theta_B\sin\phi_B - \cos\theta_B\cos\phi_B\cos\theta_A\sin\phi_A \end{pmatrix}\]
Since we are only interested in the direction of this vector and not the magnitude, we will ignore the \(r^2\)
We can then use the formula for a plane \(\mathbf{r.n}=\mathbf{a.n}\). Since this plane will go through the origin, we have \(\mathbf{r.n}=0\) and we can replace \(\mathbf{r}\) by \(\begin{pmatrix}r\cos \theta \cos \phi\\r \cos \theta\sin\phi \\ r\sin\theta\end{pmatrix}\) to get (after cancelling by \(r\))
\[(\cos \theta_A \sin \phi_A \sin\theta_B - \cos \theta_B\sin \phi_B\sin\theta_A) \cos \theta \cos \phi + (\cos\theta_B\cos\phi_B\sin\theta_A - \cos\theta_A\cos\phi_A\sin\theta_B ) \cos \theta\sin\phi + (\cos\theta_A\cos\phi_A\cos\theta_B\sin\phi_B - \cos\theta_B\cos\phi_B\cos\theta_A\sin\phi_A) \sin\theta \]
This can be simplified a little to get: