When two estimated regression lines line of Y on X and line of X on Y are perpendicular it indicates that correlation coefficient is?

This question can also be answered from a linear algebra perspective. Say you have a bunch of data points $(x,y)$. We want to find the line $y=mx+b$ that's closest to all our points (the regression line).

As an example, say we have the points $(1,2),(2,4.5),(3,6),(4,7)$. We can look at this as a simultaneous equation problem:

\begin{align} & \underline{mx + b = y}\\ & 1x + b = 2 \\ & 2x + b = 4.5 \\ & 3x + b = 6 \\ & 4x + b = 7 \end{align}

In matrix form:

$$ \left[\begin{matrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \\ 4 & 1 \end{matrix}\right] \left[\begin{matrix} x \\ b \\ \end{matrix}\right]=\left[\begin{matrix} 2 \\ 4.5 \\ 6 \\ 7 \end{matrix}\right] $$

We see right away that $\vec{y}=(2,4.5,6,7)$ (the right hand side vector) is not in the span of the columns of our matrix, meaning we will not find an $(x,b)$ to solve our system.

The closest vector to $\vec{y}$ we can find in our column space is the projection $\vec p$ of $\vec{y}$ on the column space.

If we swap out $\vec{y}$ with its projection $\vec p$ on the column space, and solve our system of equations for $\vec p$, we get the least squares solution, aka the regression line.

I.e. we can solve

$$ \left[\begin{matrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \\ 4 & 1 \end{matrix}\right] \left[\begin{matrix} x \\ b \\ \end{matrix}\right]=\left[\begin{matrix} p_1 \\ p_2 \\ p_3 \\ p_4 \end{matrix}\right] $$

to obtain the regression line $y=mx+b$ (here $m$ is the correlation coefficient normally called $\beta$).

If you did $x=my+b$ instead, you'd have:

$$ \left[\begin{matrix} 2 & 1 \\ 4.5 & 1 \\ 6 & 1 \\ 7 & 1 \end{matrix}\right] \left[\begin{matrix} y \\ b \\ \end{matrix}\right]=\left[\begin{matrix} 1 \\ 2 \\ 3 \\ 4 \end{matrix}\right] $$

To find the regression line, we'd have to solve this system using the projection $\vec r$ of $\vec x = (1,2,3,4)$ on to the column space of our new matrix.

That is, we swap $(1,2,3,4)$ with its projection $(r_1,r_2,r_3,r_4)$ on the span of $(2,4.5,6,7)$ and $(1,1,1,1)$ and solve the system. You can solve it by hand if you want to and compare it to a least squares solution found by a computer.

The idea that the regression of y given x or x given y should be the same, is equivalent to asking if $\vec p=\vec r$ in linear algebra terms.

We know that $\vec p$ is in $span (\vec x,\vec b)$ and $\vec r$ is in $span (\vec y,\vec b)$. We known that $\vec x \neq c \vec y$ since this is what motivated us to look for a regression line in the first place.

Therefore, the intersection of $span (\vec x,\vec b)$ and $span (\vec y,\vec b)$ is $c \vec b$.

So if $\vec p=\vec r$, then $\vec p=\vec r = c \vec b$.

What type of line is $c\vec b = c(1,1,1,\dots)$? On the plane, it's $y=x$. It's the line that goes out 45° from the axes of your plot.

Most of the time our regression lines will not be of the $y=x$ type. So we can see how regression is usually not symmetric.

The correlation is symmetric however. From a linear algebra perspective the correlation (aka pearson(x,y)) is $\cos(\theta)$ where $\theta$ is the angle between $\vec x$ and $\vec y$.

In the example, the correlation/pearson(x,y) is the $\cos(\theta)$ of $(1,2,3,4)$ and $(2,4.5,6,7)$.

Clearly the angle between $\vec x$ and $\vec y$ is equal to the angle between $\vec y$ and $\vec x$, so the correlation must be too.

What do you mean by regression line of Y on X and regression line of X on Y?

Is the correlation of X and Y equal to correlation of Y and X?

What would be the value of correlation coefficient between variables X and Y if two regression lines intersect each other at an angle of 90?