# Sum standard deviation vs standard error

I'm having difficulty in determining what exactly the difference is between the 2, especially when given an exercise and I have to choose which of the 2 to use. These is how my text book describes them:

Sum standard deviation

Given is a population with a normally distributed random variable $X$. When you have a sample $n$ from this population the population is:

$X_{sum} = X_1 + X_2 ... + X_n$ with

$\mu_{Xsum} = n \times \mu_x$ and $\sigma_{Xsum} = \sqrt{n} \times \sigma_x$.

Standard error

When you have a normally distributed random variable $X$ with mean $\mu_X$ and standard deviation $\sigma_X$ and sample length $n$, the sample mean $\bar{X}$ is normally distributed with $\mu_{\bar{x}} = \mu_X$ and $\sigma_{\bar{x}} = \dfrac{\sigma_X}{\sqrt{n}}$

These 2 are awefully similair to me to the point I can't at all decide which to use where. Here are the problems where I discovered I couldn't:

Problem 1

A filling machine fills bottles of lemonade. The amount is normally distributed with $\mu = 102 \space cl$.

$\sigma$ = $1.93\space cl$.

• Calculate the chance that out of 12 bottles the average volume is $100 \space cl$.

The problem itself is easy, however the troublesome part is what to choose for the standard deviation of the sample. Here they use $\dfrac{1.93}{\sqrt{12}}$ which I can live with, until I encountered the second problem.

Problem 2

A tea company puts 20 teabags in one package. The weight of a teabag is normally distributed with $\mu = 5.3 \space g$ and $\sigma = 0.5 \space g.$

• Calculate the chance that a package weighs less than 100 grams.

Here I thought they'd also use $\dfrac{0.5}{\sqrt{20}}$, but instead they use $\sqrt{20} \times 0.5$.

Can someone clear up the confusion?

• You should tag this as "homework" as well, since it seems to be a homework question. – Placidia Jan 20 '13 at 17:33
• @Placidia Are you kidding me?! This isn't homework, this is about understanding and differentiating 2 general concepts in statistics, which then could be implemented in homework questions.. like every other mathematical concept.. – JohnPhteven Jan 20 '13 at 17:36
• My textbook confused me; (Freedman, Pisani, Purves, Statistics, Fourth Edition). The chapter is titled "The Standard Error", uses $\sqrt{n} \times \sigma_x$ but the acronym is "SE", which must be a hint that my textbook is referring to "sum standard deviation"; indeed the textbook describes the experiment: "When drawing at random with replacement from a box of numbered tickets; the standard error for the sum of the draws is..." – The Red Pea Nov 7 '19 at 7:26
• ... more from the confusing part of Statistics textbook Freedman, Pisani, Purves, Fourth Edition: "In this book, we use SD for data and SE for chance quantities (random variables). This distinction is not standard and the term SD is often used in both situations" Indeed, as in this SE question, we refer to the "SD of X" ($\sigma_{X}$) and the "SD of sum of X" ($\sigma_{X_{sum}}$) – The Red Pea Nov 7 '19 at 7:37

The sum standard deviation is, as the name suggests, the standard deviation of the sum of $n$ random variables. The standard error you're talking about is just another name for the standard deviation of the mean of $n$ random variables. As you noted, the two formulas are closely related; since the sum of $n$ random variables is $n$ times the mean of $n$ random variables, the standard deviation of the sum is also $n$ times the standard deviation of the mean:

$\sigma_{X_{sum}} = \sqrt n\sigma_X = n \times \frac{\sigma_X}{\sqrt n} = n\times \sigma_\bar{X}$.

In the first problem you are dealing with a mean, the average of twelve bottles, so you use the standard deviation of the mean, which is called standard error. In the second problem you are dealing with a sum, the total weight of 20 packages, so you use the standard deviation of the sum.

Summary: use standard error when dealing with the mean (averages); use sum standard deviation when dealing with the sum (totals).

• But what I think is that theyask one about the sum of 12 bottles, and the mean of that sum? In other words, they're too similair to me.. – JohnPhteven Jan 20 '13 at 18:37
• There's no sum in question one. Each bottle is filled with an amount given by a normal distribution with mean 102, the question asks about the mean of twelve bottles. Where do you see a sum? – Jonathan Christensen Jan 20 '13 at 18:45
• Oh wait nevermind, I was being a little bit blind! In the first one they ask about the MEAN (i.e. average) out of a sample, in the second they transform a sample of 6 into '1' object (>namely, the box of teabags), with its own SD and M! – JohnPhteven Jan 20 '13 at 18:45
• I think I was writing my response the same time you were doing yours. Nice answer. – Placidia Jan 20 '13 at 18:46
• I meant 20, my brain is random, no idea how I got to 6 – JohnPhteven Jan 20 '13 at 18:48

The first standard deviation formula you gave is the SD for a sum. The standard error is the SD of the sample mean. Remember that: $\text{Var}(aX)=a^2 \text{Var}(X)$ and the variance of the sum is the sum of the variances (First formula). So

$\text{Var}(\bar{X})=\frac{n\sigma^2}{n^2}=\sigma^2/n$. Taking the square root gives the result.

Recall:

$\text{Var}(\sum X_i)=\sum (\text{Var}(X_i)=n \sigma^2.$ The Variance of the sums.

Problem 1 is looking for a statement about the sample mean; Problem 2 is about the sum, since the weight of the package is the sum of the weights of individual tea bags.

• Nice answer, +1, but I gave the other one a best answer since I read it first and it answered my question first. – JohnPhteven Jan 20 '13 at 18:48


Minimum number of iterations in Newton's method to find a square root

I am writing an algorithm that evaluates the square root of a positive real number $y$. To do this I am using the Newton-Raphton method to approximate the roots to $f(x)=x^2-y$. The $n^{th}$ iteration gives $$x_n=\frac{x_{n-1}^2+y}{2x_{n-1}}$$ as an approximation to $\sqrt{y}$. I found that starting with an initial guess $x_0=1$ works pretty well generally, so an answer to the question below that assumes $x_0=1$ is fine.

My question: is there an exact expression for the minimum $N$ of iterations needed to attain a given precision $p$ in the approximate solution $x_N$? In other words I'm looking for the smallest integer $N$ such that $$\left|\frac{x_N-\sqrt y}{\sqrt y}\right|<p.$$

I've thought about this for a while and played around with the expression for the errors $\epsilon_n = x_n - \sqrt y$ which can be shown to satisfy $\epsilon_{n+1}=\epsilon_n^2\,/\,2x_n$, but I can't find an answer. I've looked around on Google but I couldn't find an answer either.

Any pointers to a solution online or help would be much appreciated. A follow-up would of course be: can $x_0$ be optimised (while being a simple enough expression in terms of $y$) in order to minimise $N$?

• I don't have an answer to your overall question, but here's a little bit of intuition: $x_n$ is the average of $x_{n-1}$ and $y/x_{n-1}$. One of these numbers must be less than $\sqrt{y}$, and one must be greater, so the average is a reasonable update. And of course, the closer $x_0$ is to $\sqrt{y}$, the better. Something like $x_0 = \lfloor \sqrt{y}\rfloor$ is a good guess (take the biggest perfect square less than $y$, and start with its square root). – BaronVT Nov 9 '13 at 16:58
• Try analyzing $x_n^2 - y$ rather than $x_n - \sqrt{y}$. I imagine you probably want to analyze the roundoff error too, rather than assuming addition, multiplication and division are exact, which makes the problem more complicated. – Hurkyl May 2 '14 at 8:21

There is such a formula: consider

$$\frac{x_n+\sqrt y}{x_n-\sqrt y}=\frac{\frac{x_{n-1}^2+y}{2x_{n-1}}+\sqrt y}{\frac{x_{n-1}^2+y}{2x_{n-1}}-\sqrt y}=\frac{(x_{n-1}+\sqrt y)^2}{(x_{n-1}-\sqrt y)^2}=\left(\frac{x_{n-1}+\sqrt y}{x_{n-1}-\sqrt y}\right)^2.$$

By recurrence,

$$\frac{x_n+\sqrt y}{x_n-\sqrt y}=\left(\frac{x_{0}+\sqrt y}{x_{0}-\sqrt y}\right)^{2^n}.$$

If you want to achieve $2^{-b}$ relative accuracy, $x_n=(1+2^{-b})\sqrt y$,

$$2^n=\frac{\log_2\frac{(1+2^{-b})\sqrt y+\sqrt y}{(1+2^{-b})\sqrt y-\sqrt y}}{\log_2\left|\frac{x_{0}+\sqrt y}{x_{0}-\sqrt y}\right|},$$

$$n=\log_2\left(\log_2\frac{2+2^{-b}}{2^{-b}}\right)-\log_2\left(\log_2\left|\frac{x_{0}+\sqrt y}{x_{0}-\sqrt y}\right|\right).$$

The first term relates to the desired accuracy. The second is a penalty you pay for providing an inaccurate initial estimate.

If the floating-point representation of $y$ is available, a very good starting approximation is obtained by setting the mantissa to $1$ and halving the exponent (with rounding). This results in an estimate which is at worse a factor $\sqrt 2$ away from the true square root.

$$n=\log_2\left(\log_2\left(2^{b+1}+1\right)-\log_2\left(\log_2\frac{\sqrt 2+1}{\sqrt 2-1}\right)\right) \approx\log_2(b+1)-1.35.$$ In the case of single precision (23 bits mantissa), 4 iterations are always enough. For double precision (52 bits), 5 iterations.

On the opposite, if $1$ is used as a start and $y$ is much larger, $\log_2\left|\frac{1+\sqrt y}{1-\sqrt y}\right|$ is close to $\frac{2}{\ln(2)\sqrt y}$ and the formula degenerates to $$n\approx\log_2(b+1)+\log_2(\sqrt y)-1.53.$$ Quadratic convergence is lost as the second term is linear in the exponent of $y$.

If your starting value of $y$ is correctly "scaled" between $1$ and $100$, then a good initial guess is

$X_0 = 1,1545 + 0,11545*y$ (linear approx) or if you prefer a second order approx. you can use:

$X_0 = 1,0 + y*( 0,18 - 0,0009*y)$

Using $X_0$, you need only 3 iterations (Newton / Heron) to get a Relative Error less than 0,0001 % !

Now, here is a fast method to get an excellent $X_0$ (with less than 1% error):

Store a pre-calculated constants table containing all the square roots with rounded 4 decimals in the range $[1 ; 100]$. So $table(i)=sqrt(i)$.

Use Simple Linear Interpolation to get a good $X_0$. Now, the Maximum Relative Error is still greater than 1% ! ( 1.5% near $y=1.4142$).

You can simply divide this Maximum Relative Error (MRE) by a factor of 2 by doing this trick:

Simply replace the 1st entry ($\sqrt(1)=1.0000$ by $\sqrt(1)=1.0075$ ) AND the second entry ( $\sqrt(2)=1.4142$ by $\sqrt(2)=1.4248$ ). That's all. You have a MRE of 0,75%! So, the WORST case is a $X_0$ with only 0.75% in the WHOLE RANGE $[1 ; 100]$.

Unscale your number, and with only ONE Newton / Heron iteration get a 0.0028% MRE in any case, and with two iteration you have a 0.00000004% MRE!!

That's Fast, Simple and very Accurate. Have fun.


# Expected value of the distance between nodes in a binary tree

If there are 16 leaves in a full binary tree and two nodes $$a$$ and $$b$$ chosen at random, then what is the expected value of the distance between $$a$$ and $$b$$ in T?

My question here is, how do I correctly approach this question?

(Also tell me how to develop numerical skill in algorithm?)

Let's name the nodes $$\mathit{node0}$$ through $$\mathit{node15}$$. (with the implication that that's the order of the nodes)

I would approach this problem first by saying "if $$a$$ is $$\mathit{node0}$$, then what's the expected value of the distance between $$a$$ and $$b$$?" Then I'd ask the question "what if $$a$$ is $$\mathit{node1}$$?", etc. Presumably along the way there would be patterns and symmetries I could make use of so I wouldn't need to do a long calculation 16 times.

So let's take that first question: "if $$a$$ is $$\mathit{node0}$$, then what's the expected value of the distance between $$a$$ and $$b$$?"

First off, the distance between $$\mathit{node0}$$ and $$\mathit{node0}$$ is $$0$$. To $$\mathit{node1}$$, the distance is $$2$$. To nodes $$\mathit{node2}$$ or $$\mathit{node3}$$, the distance is $$4$$. To the four nodes $$\mathit{node4}$$ through $$\mathit{node7}$$, the distance is $$6$$. To the remaining eight nodes the distance is $$8$$.

So the expected distance when $$a$$ is $$\mathit{node0}$$ is $$(0 + 2 + 2*4 + 4*6 + 8*8)/16 = 6.125$$.

Now figure out the expected distance when $$a$$ is $$\mathit{node1}$$, and then other nodes. (there is an obvious pattern - prove it)

Then average over those results to find the answer.

Often, you will find a shorter or more elegant solution to a problem as you are working on it. This is normal, and should not be taken as a sign that your initial approach was wrong or that you should have seen the more elegant approach without trying your initial approach first, any more than a writer should expect to write an essay without first writing some notes or a rough draft.

Unfortunately, though writers are often trained in the idea of rough drafts and revisions, solutions to math or computer science problems are usually only presented in their final form without much indication of the bumbling around and blind corners that really went into discovering the elegant explanation.

• "To $\mathit{node1}$, the distance is $2$. To nodes $\mathit{node2}$ or $\mathit{node3}$, the distance is $4$." how?? Can u draw how node1 getting distance 2 and node2 and node3 both getting distance 4?? – Srestha Mar 14 at 16:23


# How do you calculate the expectation of $\left(\sum_{i=1}^n {X_i} \right)^2$?

If $X_i$ is exponentially distributed $(i=1,...,n)$ with parameter $\lambda$ and $X_i$'s are mutually independent, what is the expectation of

$$\left(\sum_{i=1}^n {X_i} \right)^2$$

in terms of $n$ and $\lambda$ and possibly other constants?

Note: This question has gotten a mathematical answer on http://math.stackexchange.com/q/12068/4051. The readers would take a look at it too.

-
The two copies of this question reference each other and, appropriately, the stats site (here) has a statistical answer and the math site has a mathematical answer. It seems like a good division: let it stand! – whuber Mar 4 '11 at 21:59

If $x_i \sim Exp(\lambda)$, then (under independence), $y = \sum x_i \sim Gamma(n, 1/\lambda)$, so $y$ is gamma distributed (see wikipedia). So, we just need $E[y^2]$. Since $Var[y] = E[y^2] - E[y]^2$, we know that $E[y^2] = Var[y] + E[y]^2$. Therefore, $E[y^2] = n/\lambda^2 + n^2/\lambda^2 = n(1+n)/\lambda^2$ (see wikipedia for the expectation and variance of the gamma distribution).

-
+1 Nice answer! – whuber Nov 27 '10 at 17:10
Thanks. A very neat way of answering the question (leading to the same answer) was also provided on math.stackexchange (link above in the question) a few minutes ago. – Wolfgang Nov 27 '10 at 17:24
The math answer computes the integrals using linearity of expectation. In some ways it's simpler. But I like your solution because it exploits statistical knowledge: because you know a sum of independent Exponential variables has a Gamma distribution, you're done. – whuber Nov 27 '10 at 21:19
I enjoyed it quite a bit and I am by no means a statistician or a mathematician. – Kortuk Nov 29 '10 at 18:04
very elegant answer. – Cyrus S Nov 30 '10 at 16:44

The answer above is very nice and completely answers the question but I will, instead, provide a general formula for the expected square of a sum and apply it to the specific example mentioned here.

For any set of constants $a_1, ..., a_n$ it is a fact that

$$\left( \sum_{i=1}^{n} a_i \right)^2 = \sum_{i=1}^{n} \sum_{j=1}^{n} a_{i} a_{j}$$

this is true by the Distributive property and becomes clear when you consider what you're doing when you calculate $(a_1 + ... + a_n) \cdot (a_1 + ... + a_n)$ by hand.

Therefore, for a sample of random variables $X_1, ..., X_n$, regardless of the distributions,

$$E \left( \left[ \sum_{i=1}^{n} X_i \right]^2 \right) = E \left( \sum_{i=1}^{n} \sum_{j=1}^{n} X_i X_j \right) = \sum_{i=1}^{n} \sum_{j=1}^{n} E(X_i X_j)$$

provided that these expectations exist.

In the example from the problem, $X_1, ..., X_n$ are iid ${\rm exponential}(\lambda)$ random variables, which tells us that $E(X_{i}) = 1/\lambda$ and ${\rm var}(X_i) = 1/\lambda^2$ for each $i$. By independence, for $i \neq j$, we have

$$E(X_i X_j) = E(X_i) \cdot E(X_j) = \frac{1}{\lambda^2}$$

There are $n^2 - n$ of these terms in the sum. When $i = j$, we have

$$E(X_i X_j) = E(X_{i}^{2}) = {\rm var}(X_{i}) + E(X_{i})^2 = \frac{2}{\lambda^2}$$

and there are $n$ of these term in the sum. Therefore, using the formula above,

$$E \left( \sum_{i=1}^{n} X_i \right)^2 = \sum_{i=1}^{n} \sum_{j=1}^{n} E(X_i X_j) = (n^2 - n)\cdot\frac{1}{\lambda^2} + n \cdot \frac{2}{\lambda^2} = \frac{n^2 + n}{\lambda^2}$$

-

This problem is just a special case of the much more general problem of 'moments of moments' which are usually defined in terms of power sum notation. In particular, in power sum notation:

$$s_1 = \sum_{i=1}^{n} X_i$$

Then, irrespective of the distribution, the original poster seeks $E[s_1^2]$ (provided the moments exist). Since the expectations operator is just the 1st Raw Moment, the solution is given in the mathStatica software by:

[ The '___ToRaw' means that we want the solution presented in terms of raw moments of the population (rather than say central moments or cumulants). ]

Finally, if $X$ ~ Exponential($\lambda$) with pdf $f(x)$:

f = Exp[-x/λ]/λ;      domain[f] = {x, 0, ∞} && {λ > 0};

then we can replace the moments $\mu_i$ in the general solution sol with the actual values for an Exponential random variable, like so:

All done.

P.S. The reason the other solutions posted here yield an answer with $\lambda^2$ in the denominator rather than the numerator is, of course, because they are using a different parameterisation of the Exponential distribution. Since the OP didn't state which version he was using, I decided to use the standard distribution theory textbook definition Johnson Kotz et al … just to balance things out :)

-


# Thread: Help with Probability flipping a quarter!!!!

1. ## Help with Probability flipping a quarter!!!!

So here is the problem. Maybe someone can help me. Thanks ahead of time anyone.

John and Judy play a game of flipping a quarter. If it comes up heads, John gets a point and if it comes up tails, Judy gets a point. They each bet $5 to play and the first person to get ten points wins the whole$10. At a point where John has 7 points and Judy has 5 points, the game is interrupted and they can't continue. John thinks he should win since he's ahead while Judy thinks she should win since she'd make a comeback. You are their friend, they agree to let you decide how to split up the money between them. Come up with a split that is most fair to both John and Judy.

2. Originally Posted by 2y4life
So here is the problem. Maybe someone can help me. Thanks ahead of time anyone.

John and Judy play a game of flipping a quarter. If it comes up heads, John gets a point and if it comes up tails, Judy gets a point. They each bet $5 to play and the first person to get ten points wins the whole$10. At a point where John has 7 points and Judy has 3 points, the game is interrupted and they can't continue. John thinks he should win since he's ahead while Judy thinks she should win since she'd make a comeback. You are their friend, they agree to let you decide how to split up the money between them. Come up with a split that is most fair to both John and Judy.
Calculate the probability $\displaystyle p_{John}$ that John would have won from this position.

Calculate the probability $\displaystyle p_{Judy}$ that Judy would have won from this position.

Then give John $\displaystyle \text{\$}10 \, p_{John}$and give Judy$\displaystyle \text{\$}10 \, p_{Judy}$.

3. Originally Posted by mr fantastic
Calculate the probability $\displaystyle p_{John}$ that John would have won from this position.

Calculate the probability $\displaystyle p_{Judy}$ that Judy would have won from this position.

Then give John $\displaystyle \text{\$}10 p_{John}$and give Judy$\displaystyle \text{\$}10 p_{Judy}$.
Yea...but is there a more simple way of finding that out? I tried and there are too many possibilities. John could flip another head and then flip 5 tails in a row and Judy would win or vice versa.

4. Originally Posted by 2y4life
So here is the problem. Maybe someone can help me. Thanks ahead of time anyone.

John and Judy play a game of flipping a quarter. If it comes up heads, John gets a point and if it comes up tails, Judy gets a point. They each bet $5 to play and the first person to get ten points wins the whole$10. At a point where John has 7 points and Judy has 5 points, the game is interrupted and they can't continue. John thinks he should win since he's ahead while Judy thinks she should win since she'd make a comeback. You are their friend, they agree to let you decide how to split up the money between them. Come up with a split that is most fair to both John and Judy.
This question is historically interesting ..... read this to see why.

5. Originally Posted by 2y4life
Yea...but is there a more simple way of finding that out? I tried and there are too many possibilities. John could flip another head and then flip 5 tails in a row and Judy would win or vice versa.
Let X be the random variable number of points John wins.
Let Y be the random variable number of points Judy wins.

Then:

Pr(John wins from this position) = Pr(X = 3) times Pr(Y < 5).

Pr(Judy wins from this position) = Pr(X < 3) times Pr(Y = 5).

X and Y both follow binomial distributions.

Edit: Actually it's slightly more complicated than this I just realised but I have to rush off now. I'll have more to say later unless someone else gets in first.

6. Originally Posted by mr fantastic
This question is historically interesting ..... read this to see why.
Yea, this was an extra credit problem for us to take home and figure out and my TA said that this problem came from two mathematicians named Blaise and Pierre.

I want to put this down as the answer but I'm not sure if this would constitute as a fair answer:

John- 7/12 x $10=$5.83
Judy- 5/12 x $10=$4.17

7. Originally Posted by mr fantastic
Let X be the random variable number of points John wins.
Let Y be the random variable number of points Judy wins.

Then:

Pr(John wins from this position) = Pr(X = 3) times Pr(Y < 5).

Pr(Judy wins from this position) = Pr(X < 3) times Pr(Y = 5).

X and Y both follow binomial distributions.

Edit: Actually it's slightly more complicated than this I just realised but I have to rush off now. I'll have more to say later unless someone else gets in first.
You only need to work out one of John winning or Judy winning (you know why, right).

Probability of John winning: You need to calculate Pr(X = 3) times Pr(Y < 5) for n = 3, 4, 5, 6 and 7.

Probability of Judy winning: You need to calculate Pr(X < 3) times Pr(Y = 5) for n = 5, 6 and 7.


# Dimension of vector space, countable, uncountable?

In set theory, when we talk about cardinality of a set we have notions like finite set, countably infinite and uncountably infinite sets.

Main Question

Let's talk about dimension of a vector space. In Linear Algebra I have always heard about two terms either a vector space is finite dimensional for example $\mathbb{R}^n$ or infinite dimensional for example $C[0,1]$.

Why don't we have notions like countably infinite dimensional vector space and uncountably infinite dimensional vector space.

May be, I am not able to see the bigger picture.

Extras

P.S. Long time ago, I was in a talk on Enumerative Algebraic geometry and the professor said, I always think of a positive natural number as a dimension of some vector space.

Don't we have then uncountable dimension vector space?/

• Some infinite-dimensional vector spaces have countable dimension, some have uncountable dimension. The dimension of a vector space is a well-defined cardinality. So, what's the question? – Lord Shark the Unknown Sep 13 '18 at 11:59
• I asked this as I have never heard of terms like countable dimension or uncountable dimension in books. @LordSharktheUnknown – StammeringMathematician Sep 13 '18 at 12:01
• An infinite countable dimmensionnal spaces cannot be complete but they exist. Consider the vector space of real sequences having finite support. – nicomezi Sep 13 '18 at 12:14
• "I have never heard of terms like countable dimension or uncountable dimension in books" --- This is because beginning and intermediate level linear algebra books rarely distinguish more precisely than "finite dimension" and "infinite dimension". It's usually only in graduate level algebra courses (e.g. probably all of the "third level" books in my answer to High-level linear algebra book) where you'll find the various notions of "algebraic dimension" defined as a cardinal number. – Dave L. Renfro Sep 13 '18 at 12:31
• Just because you don't see it in your text doesn't mean it doesn't exist. You're quite right that it makes sense! – Noah Schweber Sep 13 '18 at 12:31

We do have the notions of countable/uncountable dimensions. Just as a set can be finite or infinite (without specifying which infinite cardinality the set as) a vector space can be finite dimensional or infinite dimensional. We can then go one step more and ask, if the dimension is infinite, which infinite cardinal is it?

The definition of dimension of a vector space is the cardinality of a basis for that vector space (it does not matter which basis you take, because they all have the same cardinality). Then for any cardinal number $\gamma$, you can have a vector space with that dimension. For example, if $\Gamma$ is a set with cardinality $\gamma$, let $c_{00}(\Gamma)$ be the space of all $\mathbb{F}$-valued functions $f$ such that $$\text{supp}(f)=\{x\in \Gamma: f(x)\neq 0\}$$ is finite. Then let $\delta_x\in c_{00}(\Gamma)$ be the function such that $\delta_x(y)=0$ if $y\neq x$ and $\delta_x(y)=1$ if $y=x$. Then $(\delta_x)_{x\in \Gamma}$ is a basis for $c_{00}(\Gamma)$ with cardinality $\gamma$. If $\Gamma=\mathbb{N}$, we have a vector space with countably infinite dimension. If $\Gamma=\mathbb{R}$, we have a vector space with dimension equal to the cardinality of the continuum.

However, for infinite dimensional topological vector spaces (and for infinite dimensional Hilbert and Banach spaces in particular) the usual notion of a basis of limited use. This is because the coordinate functionals for an infinite basis do not interact very well with the topology (one can show that if $(e_i, e^*_i)_{i\in I}$ is a basis together with its coordinate functionals for an infinite dimensional Banach space, then only finitely many of the functionals $e^*_i$ can be continuous). Since the notion of a basis is not as useful in the infinite dimensional topological space case as it is in the finite dimensional case, you can see less emphasis on what the exact dimension is in this case.

However, in this situation you get into discussions of other types of coordinate systems (such as Schauder bases, FDDs, unconditional bases, etc.), which are different from the notion of an (algebraic) basis. You also can ask about density character instead of dimension, which is the smallest cardinality of a dense subset. This encodes topological information, while the purely algebraic notion of a basis does not. For example, infinite dimensional Hilbert space $\ell_2$ has no countable basis, it does have a countable, dense subset. So the dimension is that of the continuum, but the density character is $\aleph_0$.

The dimension of a vector space is the cardinality of a basis for that vector space. To say that a vector space has finite dimension therefore means that the cardinality of a basis for that vector space is finite. Since finite cardinalities are the same thing as natural numbers, we are safe in saying, for finite dimensional vector spaces, that the dimension is a natural number.

In general, some sets are countably infinite and some sets are uncountably infinite. So, applying this to those sets which happen to be bases of vector spaces, some vector spaces have countably infinite bases and therefore countably infinite dimension, and other vector space have uncountable infinite bases and therefore uncountably infinite dimension.

An example of a vector space over $\mathbb R$ of countably infinite dimension is $\mathbb R^{\infty}$ which is the space of infinite sequences of real numbers such that all but finitely terms in the sequence are equal to $0$. A countably infinite basis consists of $(1,0,0,0,...)$, $(0,1,0,0,...)$, $(0,0,1,0,...)$ and so on.

An example of a vector space over $\mathbb R$ of uncountably infinite dimension is the one you mention in your question, $C[0,1]$.

• That's okay. I do often write things wrong the first time. – Lee Mosher Sep 13 '18 at 12:18
• ... all (but finitely many) terms are zero. – Lee Mosher Sep 13 '18 at 12:19
• Had a hard time understanding your sentence, sorry for incovenience. – nicomezi Sep 13 '18 at 12:22

For the sake of a real world example: The electron in a hydrogen atom can take on countably many states. Each state is a basis vector for the span of possible electron states of hydrogen. There's are bound states and bound states are typically discrete. The relevant Schrodinger Equation also permits scattering states which have a continuous spectrum of possible energy states implying an uncountable vector space spanning the possible scattering states.

The following theorem is an example where we need to distinguish between countably/uncountably infinite dimensional vector spaces. Some important theorems, such as Hilbert Nullstallensatz, can be deduced from it.

Let $$A$$ be an associative, not necessarily commutative, $$\mathbb{C}-$$algebra with unit. For $$a \in A$$ define $$\text{Spec } a = \{\lambda \in \mathbb{C} | a-\lambda \text{ is not invertible}\}$$ Assume that $$A$$ has no more than countable dimension over $$\mathbb{C}$$. Then

(a) If $$A$$ is a division algebra, then $$A=\mathbb{C}$$

(b) For all $$a \in A$$ we have $$\text{Spec } A \neq \emptyset$$; furthermore, $$a \in A$$ is nilpotent if and only if $$\text{Spec } A = \{0\}$$

(Adapted from Representation Theory and Complex Geometry by Chriss and Ginzburg, theorem 2.1.1.)

The proof uses the fact that for any $$a \in A$$, $$\{(a - \lambda)^{-1} | \lambda \in \mathbb{C}\}$$ is an uncountable family of elements of $$A$$. But $$A$$ has only countable dimension over $$\mathbb{C}$$, so they are not linearly indenpendent over $$\mathbb{C}$$.

A weak version of Hilbert Nullstellensatz:

Let $$A$$ be a finitely generated commutative algebra over $$\mathbb{C}$$. Then any maximal ideal of $$A$$ is the kernel of an algebra homomorphism $$A \to \mathbb{C}$$

This follows directly from the above theorem: $$A$$ finitely generated $$\Longrightarrow$$ $$A$$ has countable dimension over $$\mathbb{C}$$ $$\Longrightarrow$$ $$A/\mathfrak{m}$$ has countable dimension over $$\mathbb{C}$$ $$\Longrightarrow$$ $$A/\mathfrak{m}=\mathbb{C}$$

(We can also deduce the strong version of Nullstellensatz from it but need more argument.)

• Welcome the Mathematics Stack Exchange! A quick tour of the site (math.stackexchange.com/tour) will help you get the most of your time here. – dantopa Dec 28 '18 at 4:57


## anonymous one year ago HELP ME PLEASE Which of the following exponential functions goes through the points (1, 6) and (2, 12)? f(x) = 3(2)x f(x) = 2(3)x f(x) = 3(2)−x f(x) = 2(3)−x

1. ybarrap

Plug in x=1 and see if it equals 6 Plug in x=2 and see if it equals 12 Make sense?

2. anonymous

not really? can you guide me through it? (xD I dont want to be an answer-hogger/wanter.. I dont want the answer, just explanations :) ) @freckles

3. anonymous

Do you understand that coordinates are in the form (x,y)?

4. anonymous

yes. @BMF96

5. anonymous

f(x) = y

6. anonymous

What don't you understand?

7. jim_thompson5910

If $\Large f(x) = 7(3)^x$ (for example), then what is the value of f(x) when x = 2? In other words, what is f(2) equal to?

8. anonymous

f(2)=441?? #CONFUSED

9. jim_thompson5910

Replace each x with 2 $\Large f(x) = 7(3)^x$ $\Large f(2) = 7(3)^2$ $\Large f(2) = 7(9)$ $\Large f(2) = 63$ Do you see how I got f(2) to be equal to 63?

10. jim_thompson5910

btw you square first and then you multiply

11. anonymous

ok, I forgot that rule :) (like DUHR, Bella, get a grip)

12. jim_thompson5910

Since $$\Large f({\color{red}{2}}) = {\color{blue}{63}}$$ from my example, this means the point $$\Large ({\color{red}{x}},{\color{blue}{y}})=({\color{red}{2}},{\color{blue}{63}})$$ lies on the function curve of f(x)

13. anonymous

okayyy....

14. anonymous

$$f(x)=y= \begin{cases} 3(2)^x\\ 2(3)^x\\ 3(2)^{-x}\\ 2(3)^{-x} \end{cases}\qquad \qquad \begin{array}{llll} x&y \\\hline\\ {\color{brown}{ 1}}&3(2)^{\color{brown}{ 1}}\\ &2(3)^{\color{brown}{ 1}}\\ &3(2)^{-{\color{brown}{ 1}}}\\ &2(3)^{-{\color{brown}{ 1}}}\\ {\color{brown}{ 2}}&3(2)^{\color{brown}{ 2}}\\ &2(3)^{\color{brown}{ 2}}\\ &3(2)^{-{\color{brown}{ 2}}}\\ &2(3)^{-{\color{brown}{ 2}}} \end{array}$$

15. jim_thompson5910

so what ybarrap said at the top, you plug in each x coordinate into each function and see if you get the correct corresponding y coordinates Let's say we pick choice B at random $\Large f(x) = 2(3)^x$ The first point is (1,6). To test if this point lies on the function f(x) curve, we plug in x = 1 and see if y = 6 pops out $\Large f(x) = 2(3)^x$ $\Large f(1) = 2(3)^1$ $\Large f(1) = 2(3)$ $\Large f(1) = 6$ It does, so (1,6) is definitely on this curve. How about (2,12)? Let's check $\Large f(x) = 2(3)^x$ $\Large f(2) = 2(3)^2$ $\Large f(2) = 2(9)$ $\Large f(2) = 18$ Nope. The point (2,18) actually lies on this function curve and NOT (2,12). So we can rule out choice B.

16. jim_thompson5910

So what just happened was that I've proven that the function $$\Large f(x) = 2(3)^x$$ does NOT go through both points (1,6) and (2,12).

17. anonymous

okay, so its A, C, or D lol.... so lets try to rule out A...

18. anonymous

@jim_thompson5910

19. jim_thompson5910

what did you get so far in checking choice A?

20. anonymous

that A is, in fact NOT the answer!?

21. jim_thompson5910

if x = 1, then what is f(1) ?

22. anonymous

f

23. anonymous

Wait, so it IS A!!!!!

24. jim_thompson5910

$\Large f(x) = 3(2)^x$ $\Large f(1) = 3(2)^1$ $\Large f(1) = \underline{ \ \ \ \ \ \ \ } \text{ (fill in the blank)}$

25. anonymous

6

26. jim_thompson5910

27. anonymous

12

28. jim_thompson5910

Good. Choice A is definitely the answer. As practice, why not go through C and D and eliminate them. With choice C, if x = 1, then what is f(x) equal to?


# Simulating random variables from a discrete distribution

I have the following discrete distribution where $p$ is a known constant:

$p(x,p)= \frac{(1-p)^3}{p(1+p)}x^2p^x , (0<p<1), x=0, 1, 2, \ldots$ .

How can I sample from this distribution?

– Tim
May 23, 2016 at 15:03
• Does this even sum to 1 for some value of $p$? (I doubt so) May 23, 2016 at 15:19
• @Elvis It does sum to 1. May 23, 2016 at 15:41
• @MatthewGunn My bad. May 24, 2016 at 6:43

This answer develops a simple procedure to generate values from this distribution. It illustrates the procedure, analyzes its scope of application (that is, for which $p$ it might be considered a practical method), and provides executable code.

### The Idea

Because

$$x^2 = 2\binom{x}{2} + \binom{x}{1},$$

consider the distributions $f_{p;m}$ given by

$$f_{p;m}(x) \propto \binom{x}{m-1}p^x$$

for $m=3$ and $m=2$.

A recent thread on inverse sampling demonstrates that these distributions count the number of observations of independent Bernoulli$(1-p)$ variables needed before first seeing $m$ successes, with $x+1$ equal to that number. It also shows that the normalizing constant is

$$C(p;m)=\sum_{x=m-1}^\infty \binom{x}{m-1}p^x = \frac{p^{m-1}}{(1-p)^m}.$$

Consider the probabilities in the question,

$$x^2 p^x = \left( 2\binom{x}{2} + \binom{x}{1} \right)p^x = 2 \binom{x}{2}p^x + \binom{x}{1} p^x =2 C(p;3) f_{p;3}(x) + C(p;2) f_{p;2}(x).$$

Consequently, the given distribution is a mixture of $f_{p;3}$ and $f_{p;2}$. The proportions are as $$2C(p;3):C(p;2) = 2p:(1-p).$$ It is simple to sample from a mixture: generate an independent uniform variate $u$ and draw $x$ from $f_{p;2}$ when $u \lt (1-p)/(2p+1-p)$; that is, when $u(1+p) \lt 1-p$, and otherwise draw $x$ from $f_{p;3}$.

(It is evident that this method generalizes: many probability distributions where the chance of $x$ is of the form $P(x)p^x$ for a polynomial $P$, such as $P(x)=x^2$ here, can be expressed as a mixture of these inverse-sampling distributions.)

### The Algorithm

These considerations lead to the following simple algorithm to generate one realization of the desired distribution:

Let U ~ Uniform(0,1+p)
If (U < 1-p) then m = 2 else m = 3
x = 0
While (m > 0) {
x = x + 1
Let Z ~ Bernoulli(1-p)
m = m - Z
}
Return x-1

These histograms show simulations (based on 100,000 iterations) and the true distribution for a range of values of $p$.

### Analysis

How efficient is this? The expectation of $x+1$ under the distribution $f_{p;m}$ is readily computed; it equals $m/(1-p)$. Therefore the expected number of trials (that is, values of Z to generate in the algorithm) is

$$\left((1-p) \frac{2}{1-p} + (2p) \frac{3}{1-p}\right) / (1-p+2p) = 2 \frac{1+2p}{1-p^2}.$$

Add one more for generating U. The total is close to $3$ for small values of $p$. As $p$ approaches $1$, this count asymptotically is

$$1 + 2\frac{1 + 2p}{(1-p)(1+p)} \approx \frac{3}{1-p}.$$

This shows us that the algorithm will, on the average, be reasonably quick for $p \lt 2/3$ (taking up to ten easy steps) and not too bad for $p \lt 0.97$ (taking under a hundred steps).

### Code

Here is the R code used to implement the algorithm and produce the figures. A $\chi^2$ test will show that the simulated results do not differ significantly from the expected frequencies.

sample <- function(p) {
m <- ifelse(runif(1, max=1+p) < 1-p, 2, 3)
x <- 0
while (m > 0) {
x <- x + 1
m <- m - (runif(1) > p)
}
return(x-1)
}
n <- 1e5
set.seed(17)
par(mfcol=c(2,3))
for (p in c(1/5, 1/2, 9/10)) {

# Simulate and summarize.
x <- replicate(n, sample(p))
y <- table(x)

# Compute the true distribution for comparison.
k <- as.numeric(names(y))
theta <- sapply(k, function(i) i^2 * p^i) * (1-p)^3 / (p^2 + p)
names(theta) <- names(y)

# Plot both.
barplot(y/n, main=paste("Simulation for", format(p, digits=2)),
border="#00000010")
barplot(theta, main=paste("Distribution for", format(p, digits=2)),
border="#00000010")
}

@dsaxton's approach is known as inverse transform sampling and is probably the way to go for a problem like this. To be a bit more explicit, the approach is:

1. Draw $u$ from uniform distribution on (0,1).
2. Compute $x = F^{-1}(u)$ where $F^{-1}$ is the inverse of the cumulative distribution function.

Computing $x = F^{-1}(u)$ is equivalent to finding the integer $x$ that is the solution to: $$\text{minimize} \quad x \quad \text{subject to} \sum_{j=0}^x \frac{(1 - p)^3}{p(1+p)} j^2p^j \geq u$$

Quick pseudo code to do this numerically:

1. Construct a vector $\boldsymbol{m}$ such that $m_j = \frac{(1 - p)^3}{p(1+p)} j^2p^j$.
2. Create a vector $\boldsymbol{c}$ such that $c_j = \sum_{k=0}^j m_j$.
3. Find the minimum index $x$ such that $c_x \geq u$.

Draw $u$ from a uniform$(0, 1)$ distribution and let $x$ be the smallest value of $k$ for which $\sum_{j=0}^{k} \frac{(1 - p)^3}{p (1 + p)} j^2 p^j > u$. Then $x$ will be a realization from the desired distribution.

• May be clearer to write $x = \min k \text{ subject to } \sum_j^k \ldots > u$. As written, it's not terribly clear what is being minimized and that $x$ equals the optimal $k$. May 23, 2016 at 16:10


# Notation for cartesian product except one set?

Let's say I have a list of sets $S_i$, for $i=1,\ldots,n$. We often write the cartesian product of all these sets, with the exception of $S_k$ as:

$$S=S_1\times\cdots\times S_{k-1}\times S_{k+1}\times\cdots\times S_n$$

Is there a more succinct way to write it?

In Wikipedia (https://en.wikipedia.org/wiki/Cartesian_product), I found something, which might be what you are looking for: $\prod_{n=1}^k \Bbb{R} = \Bbb{R}\times \Bbb{R} \times\cdots\times \Bbb{R} = \Bbb{R}^k$. So maybe something like this one is also valid: $$\prod_{\scriptstyle i = 1\atop\scriptstyle i \ne k}^nS_i$$

where $S_i$ is the $i^\text{th}$ set of the list you mentioned.

• Possibly \prod_{i = 1\atop i \ne k}^n would be better? (with \atop separating lines instead of a comma) – CiaPan Dec 14 '17 at 11:15
• Yes, you are right. Thank you for the formatting :) – ArsenBerk Dec 14 '17 at 11:16
• I changed $\Bbb{R}$ x $\Bbb{R}$ x ... x $\Bbb{R}$ to $\Bbb{R} \times \Bbb{R} \times \cdots \times \Bbb{R}.$ That is proper MathJax usage. $\qquad$ – Michael Hardy Dec 14 '17 at 13:13
• In addition to the atop business, another way to notate the indices would be with $\in$. I could see that being especially convenient if these indices are already in a set or are repeated throughout the work. If $E=\{1,\cdots,n\}\setminus\{k\}$ then you can have $$\prod_{i\in E}S_i$$ – gen-ℤ ready to perish Dec 14 '17 at 13:42
• Another formatting comment: if you look carefully, your subscripts on the product symbol are in a smaller font than the superscript. IMHO it's better to avoid this by using \prod_{\scriptstyle i = 1\atop\scriptstyle i \ne k}^n: compare$$\prod_{i = 1\atop i \ne k}^nS_i\quad\hbox{and}\quad \prod_{\scriptstyle i = 1\atop\scriptstyle i \ne k}^nS_i$$ – David Dec 15 '17 at 1:36

I have seen a notation for this kind of construction during some of my math lectures (but can't find a reference right now). This was mostly in the context of differential forms (e.g. interior product with vector), but can be applied to your case: $$S_1\times \dotsm \times \widehat{S_k} \times\dotsm \times S_n := S_1\times \dotsm \times S_{k-1}\times S_{k+1} \times \dotsm S_n$$ The hat denotes the factor to be omitted. Note that this is not a universally standard notation, so even the professors that used it defined it at some point early in the lecture.

• This is what I would use, along with a parenthetical remark along the lines of "where the hat denotes the factor to be omitted." I very seldom see anything as formal as in the other answers. – Matthew Leingang Dec 14 '17 at 15:00
• Hatcher uses a notation similar to this in his text Algebraic Topology (see page 105 for an example). – Xander Henderson Dec 17 '17 at 2:52

In general, we can write

$$S_1 \times \dots \times S_n := \prod_{i=1}^n S_i$$

and then we can apply all conventions we are used to.

As for your question, this can be written as:

$$\prod_{i = 1 \atop i \neq k}^n S_i$$

Although it might not be common in set theory, it is common for game theorists to write $S_{-i}$ for $S_1 \times \cdots \times S_{i-1} \times S_{i+1} \times \cdots \times S_n$. See page 15 of chapter one of Osborne and Rubinstein's text on game theory, for example.

That notation is useful in game theory because, if $S_j$ represents the set of strategies available to player $j$, then one often needs to describe how all players except player $i$ have acted. Such a description will be a member of $S_i$. In particular, the notation becomes useful in defining a Nash equilibrium. See https://en.wikipedia.org/wiki/Nash_equilibrium.


2017) and the quantum hierarchical clustering algorithm based on quantum Euclidean estimator (Kong, Lai, and Xiong 2017) has been implemented. Applying the formula given above we get that: \begin{align} d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{v} \| \\ d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{w} +\vec{w} - \vec{v} \| \\ d(\vec{u}, \vec{v}) = \| (\vec{u} - \vec{w}) + (\vec{w} - \vec{v}) \| \\ d(\vec{u}, \vec{v}) \leq || (\vec{u} - \vec{w}) || + || (\vec{w} - \vec{v}) \| \\ d(\vec{u}, \vec{v}) \leq d(\vec{u}, \vec{w}) + d(\vec{w}, \vec{v}) \quad \blacksquare \end{align}, \begin{align} d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{v} \| = \sqrt{(2-1)^2 + (3+2)^2 + (4-1)^2 + (2-3)^2} \\ d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{v} \| = \sqrt{1 + 25 + 9 + 1} \\ d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{v} \| = \sqrt{36} \\ d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{v} \| = 6 \end{align}, Unless otherwise stated, the content of this page is licensed under. . The reason for this is because whatever the values of the variables for each individual, the standardized values are always equal to 0.707106781 ! The Euclidean distance between two random points [ x 1 , x 2 , . Determine the Euclidean distance between. The Pythagorean Theorem can be used to calculate the distance between two points, as shown in the figure below. The distance between two vectors v and w is the length of the difference vector v - w. There are many different distance functions that you will encounter in the world. In mathematics, the Euclidean distance between two points in Euclidean space is the length of a line segment between the two points. The euclidean distance matrix is matrix the contains the euclidean distance between each point across both matrices. A generalized term for the Euclidean norm is the L2 norm or L2 distance. Append content without editing the whole page source. u = < -2 , 3> . Solution. Wikidot.com Terms of Service - what you can, what you should not etc. The Euclidean distance between two points in either the plane or 3-dimensional space measures the length of a segment connecting the twoÂ  In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" straight-line distance between two points in Euclidean space. Otherwise, columns that have large values will dominate the distance measure. pdist2 is an alias for distmat, while pdist(X) is … You are most likely to use Euclidean distance when calculating the distance between two rows of data that have numerical values, such a floating point or integer values. If you want to discuss contents of this page - this is the easiest way to do it. If columns have values with differing scales, it is common to normalize or standardize the numerical values across all columns prior to calculating the Euclidean distance. To calculate the Euclidean distance between two vectors in Python, we can use the numpy.linalg.norm function: Applying the formula given above we get that: (2) \begin {align} d (\vec {u}, \vec {v}) = \| \vec {u} - \vec {v} \| = \sqrt { (2-1)^2 + (3+2)^2 + (4-1)^2 + (2-3)^2} \\ d (\vec {u}, \vec {v}) = \| \vec {u} - \vec {v} \| = \sqrt {1 + 25 + 9 + 1} \\ d (\vec {u}, \vec {v}) = \| \vec {u} - \vec {v} \| = \sqrt {36} \\ d (\vec {u}, \vec {v}) = \| \vec {u} - \vec {v} \| = 6 … Find out what you can do. 1 Suppose that d is very large. ml-distance-euclidean. Check out how this page has evolved in the past. By using this metric, you can get a sense of how similar two documents or words are. Computes the Euclidean distance between a pair of numeric vectors. View wiki source for this page without editing. A generalized term for the Euclidean norm is the L2 norm or L2 distance. Recall that the squared Euclidean distance between any two vectors a and b is simply the sum of the square component-wise differences. View and manage file attachments for this page. With this distance, Euclidean space becomes a metric space. Older literature refers to the metric as the Pythagorean metric. $d(\vec{u}, \vec{v}) = \| \vec{u} - \vec{v} \| = \sqrt{(u_1 - v_1)^2 + (u_2 - v_2)^2 ... (u_n - v_n)^2}$, $d(\vec{u}, \vec{v}) = d(\vec{v}, \vec{u})$, $d(\vec{u}, \vec{v}) = || \vec{u} - \vec{v} || = \sqrt{(u_1 - v_1)^2 + (u_2 - v_2)^2 ... (u_n - v_n)^2}$, $d(\vec{v}, \vec{u}) = || \vec{v} - \vec{u} || = \sqrt{(v_1 - u_1)^2 + (v_2 - u_2)^2 ... (v_n - u_n)^2}$, $(u_i - v_i)^2 = u_i^2 - 2u_iv_i + v_i^2 = v_i^2 - 2u_iv_i + 2u_i^2 = (v_i - u_i)^2$, $\vec{u}, \vec{v}, \vec{w} \in \mathbb{R}^n$, $d(\vec{u}, \vec{v}) \leq d(\vec{u}, \vec{w}) + d(\vec{w}, \vec{v})$, Creative Commons Attribution-ShareAlike 3.0 License. The primary takeaways here are that the Euclidean distance is basically the length of the straight line that's connects two vectors. 3.8 Digression on Length and Distance in Vector Spaces. Using our above cluster example, we’re going to calculate the adjusted distance between a … Euclidean metric is the “ordinary” straight-line distance between two points. Euclidean distance between two vectors, or between column vectors of two matrices. , y d ] is radicaltp radicalvertex radicalvertex radicalbt d summationdisplay i =1 ( x i − y i ) 2 Here, each x i and y i is a random variable chosen uniformly in the range 0 to 1. This victory. The corresponding loss function is the squared error loss (SEL), and places progressively greater weight on larger errors. The shortest path distance is a straight line. And now we can take the norm. Computes the Euclidean distance between a pair of numeric vectors. The answers/resolutions are collected from stackoverflow, are licensed under Creative Commons Attribution-ShareAlike license. We will now look at some properties of the distance between points in $\mathbb{R}^n$. It corresponds to the L2-norm of the difference between the two vectors. The Euclidean distance between 1-D arrays u and v, is defined as maximum: Maximum distance between two components of x and y (supremum norm) manhattan: Absolute distance between the two vectors (1 … I have the two image values G= [1x72] and G1 = [1x72]. And these is the square root off 14. A little confusing if you're new to this idea, but it … D = √ [ ( X2-X1)^2 + (Y2-Y1)^2) Where D is the distance. We determine the distance between the two vectors. if p = (p1, p2) and q = (q1, q2) then the distance is given by. u of the two vectors. Notify administrators if there is objectionable content in this page. Solution to example 1: v . is: Deriving the Euclidean distance between two data points involves computing the square root of the sum of the squares of the differences between corresponding values. ... Percentile. u, is v . Let’s assume OA, OB and OC are three vectors as illustrated in the figure 1. It can be computed as: A vector space where Euclidean distances can be measured, such as , , , is called a Euclidean vector space. General Wikidot.com documentation and help section. The squared Euclidean distance is therefore d(xÂ  SquaredEuclideanDistance is equivalent to the squared Norm of a difference: The square root of SquaredEuclideanDistance is EuclideanDistance : Variance as a SquaredEuclideanDistance from the Mean : Euclidean distance, Euclidean distance. . So there is a bias towards the integer element. The following formula is used to calculate the euclidean distance between points. View/set parent page (used for creating breadcrumbs and structured layout). Two squared, lost three square until as one. $\vec {u} = (2, 3, 4, 2)$. I need to calculate the two image distance value. The distance between two points is the length of the path connecting them. Source: R/L2_Distance.R Quickly calculates and returns the Euclidean distances between m vectors in one set and n vectors in another. {\displaystyle \left\|\mathbf {a} \right\|= {\sqrt {a_ {1}^ {2}+a_ {2}^ {2}+a_ {3}^ {2}}}} which is a consequence of the Pythagorean theorem since the basis vectors e1, e2, e3 are orthogonal unit vectors. Most vector spaces in machine learning belong to this category. Accepted Answer: Jan Euclidean distance of two vector. By using this formula as distance, Euclidean space becomes a metric space. $\vec {v} = (1, -2, 1, 3)$. Each set of vectors is given as the columns of a matrix. Installation $npm install ml-distance-euclidean. Euclidean distance.$\begingroup$Even in infinitely many dimensions, any two vectors determine a subspace of dimension at most$2$: therefore the (Euclidean) relationships that hold in two dimensions among pairs of vectors hold entirely without any change at all in any number of higher dimensions, too. Squared Euclidean Distance, Let x,yâRn. This system utilizes Locality sensitive hashing (LSH) [50] for efficient visual feature matching.$\endgroup$– whuber ♦ Oct 2 '13 at 15:23 The formula for this distance between a point X ( X 1 , X 2 , etc.) In simple terms, Euclidean distance is the shortest between the 2 points irrespective of the dimensions. Y = cdist(XA, XB, 'sqeuclidean') The Euclidean distance between two vectors, A and B, is calculated as: Euclidean distance = √ Σ(A i-B i) 2. You want to find the Euclidean distance between two vectors. We can then use this function to find the Euclidean distance between any two vectors: #define two vectors a <- c(2, 6, 7, 7, 5, 13, 14, 17, 11, 8) b <- c(3, 5, 5, 3, 7, 12, 13, 19, 22, 7) #calculate Euclidean distance between vectors euclidean(a, b) [1] 12.40967 The Euclidean distance between the two vectors turns out to be 12.40967. Computing the Distance Between Two Vectors Problem. Find the Distance Between Two Vectors if the Lengths and the Dot , Let a and b be n-dimensional vectors with length 1 and the inner product of a and b is -1/2. — Page 135, D… w 1 = [ 1 + i 1 â i 0], w 2 = [ â i 0 2 â i], w 3 = [ 2 + i 1 â 3 i 2 i]. In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" distance between twoÂ (geometry) The distance between two points defined as the square root of the sum of the squares of the differences between the corresponding coordinates of the points; for example, in two-dimensional Euclidean geometry, the Euclidean distance between two points a = (a x, a y) and b = (b x, b y) is defined as: What does euclidean distance mean?, In the spatial power covariance structure, unequal spacing is measured by the Euclidean distance d â¢ j j â² , defined as the absolute difference between twoÂ In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" distance between two points that one would measure with a ruler, and is given by the Pythagorean formula. So the norm of the vector to three minus one is just the square root off. = v1 u1 + v2 u2 NOTE that the result of the dot product is a scalar. It is the most obvious way of representing distance between two points. We here use "Euclidean Distance" in which we have the Pythagorean theorem. The result is a positive distance value. (we are skipping the last step, taking the square root, just to make the examples easy) ‖ a ‖ = a 1 2 + a 2 2 + a 3 2. Euclidean distance. We will derive some special properties of distance in Euclidean n-space thusly. The associated norm is called the Euclidean norm. The average distance between a pair of points is 1/3. First, determine the coordinates of point 1. Active 1 year, 1 month ago. And that to get the Euclidean distance, you have to calculate the norm of the difference between the vectors that you are comparing. . Definition of normalized Euclidean distance, According to Wolfram Alpha, and the following answer from cross validated, the normalized Eucledean distance is defined by: enter imageÂ In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" straight-line distance between two points in Euclidean space. This is helpfulÂ variables, the normalized Euclidean distance would be 31.627. So this is the distance between these two vectors. Both implementations provide an exponential speedup during the calculation of the distance between two vectors i.e. These names come from the ancient Greek mathematicians Euclid and Pythagoras, although Euclid did not represent distances as numbers, and the connection from the Pythagorean theorem to distance calculation wa Y1 and Y2 are the y-coordinates. Copyright ©document.write(new Date().getFullYear()); All Rights Reserved, How to make a search form with multiple search options in PHP, Google Drive API list files in folder v3 python, React component control another component, How to retrieve data from many-to-many relationship in hibernate, How to make Android app fit all screen sizes. Discussion. Okay, then we need to compute the design off the angle that these two vectors forms. Given some vectors$\vec{u}, \vec{v} \in \mathbb{R}^n$, we denote the distance between those two points in the following manner. Euclidean Distance Formula. <4 , 6>. sample 20 1 0 0 0 1 0 1 0 1 0 0 1 0 0 The squared Euclidean distance sums the squared differences between these two vectors: if there is an agreement (there are two matches in this example) there is zero sum of squared differences, but if there is a discrepancy there are two differences, +1 and –1, which give a sum of squares of 2. X1 and X2 are the x-coordinates. The Euclidean distance between two points in either the plane or 3-dimensional space measures the length of a segment connecting the two points. With this distance, Euclidean space becomes a metric space. Watch headings for an "edit" link when available. As such, it is also known as the Euclidean norm as it is calculated as the Euclidean distance from the origin. Euclidean Distance. Click here to edit contents of this page. For three dimension 1, formula is. Determine the Euclidean distance between$\vec{u} = (2, 3, 4, 2)$and$\vec{v} = (1, -2, 1, 3)$. This process is used to normalize the featuresÂ Now I would like to compute the euclidean distance between x and y. I think the integer element is a problem because all other elements can get very close but the integer element has always spacings of ones. Brief review of Euclidean distance. Euclidean distance, Euclidean distances, which coincide with our most basic physical idea of squared distance between two vectors x = [ x1 x2 ] and y = [ y1 y2 ] is the sum ofÂ The Euclidean distance function measures the âas-the-crow-fliesâ distance. In this article to find the Euclidean distance, we will use the NumPy library. Change the name (also URL address, possibly the category) of the page. Basic Examples (2) Euclidean distance between two vectors: Euclidean distance between numeric vectors: If not passed, it is automatically computed. Euclidean distance Suppose w 4 is [â¦] Construction of a Symmetric Matrix whose Inverse Matrix is Itself Let v be a nonzero vector in R n . The associated norm is called the Euclidean norm. Usage EuclideanDistance(x, y) Arguments x. Numeric vector containing the first time series. Glossary, Freebase(1.00 / 1 vote)Rate this definition: Euclidean distance. Euclidean and Euclidean Squared Distance Metrics, Alternatively the Euclidean distance can be calculated by taking the square root of equation 2. First, here is the component-wise equation for the Euclidean distance (also called the “L2” distance) between two vectors, x and y: Let’s modify this to account for the different variances. The standardized Euclidean distance between two n-vectors u and v is $\sqrt{\sum {(u_i-v_i)^2 / V[x_i]}}.$ V is the variance vector; V[i] is the variance computed over all the i’th components of the points. Let’s discuss a few ways to find Euclidean distance by NumPy library. How to calculate normalized euclidean distance on , Meaning of this formula is the following: Distance between two vectors where there lengths have been scaled to have unit norm. It can be calculated from the Cartesian coordinates of the points using the Pythagorean theorem, therefore occasionally being called the Pythagorean distance. u = < v1 , v2 > . See pages that link to and include this page. and. This library used for manipulating multidimensional array in a very efficient way. Available distance measures are (written for two vectors x and y): euclidean: Usual distance between the two vectors (2 norm aka L_2), sqrt(sum((x_i - y_i)^2)). and a point Y ( Y 1 , Y 2 , etc.) their Dot Product of Two Vectors The dot product of two vectors v = < v1 , v2 > and u = denoted v . I've been reading that the Euclidean distance between two points, and the dot product of theÂ Dot Product, Lengths, and Distances of Complex Vectors For this problem, use the complex vectors. In a 3 dimensional plane, the distance between points (X 1 , … Sometimes we will want to calculate the distance between two vectors or points. The points are arranged as m n -dimensional row vectors in the matrix X. Y = cdist (XA, XB, 'minkowski', p) (Zhou et al. gives the Euclidean distance between vectors u and v. Details. Compute the euclidean distance between two vectors. Understand normalized squared euclidean distance?, Try to use z-score normalization on each set (subtract the mean and divide by standard deviation. Ask Question Asked 1 year, 1 month ago. Before using various cluster programs, the proper data treatment isâÂ Squared Euclidean distance is of central importance in estimating parameters of statistical models, where it is used in the method of least squares, a standard approach to regression analysis. Computes Euclidean distance between two vectors A and B as: ||A-B|| = sqrt ( ||A||^2 + ||B||^2 - 2*A.B ) and vectorizes to rows of two matrices (or vectors). With this distance, Euclidean space becomes a metric space. Older literature refers to the metric as the Pythagorean metric. Something does not work as expected? Example 1: Vectors v and u are given by their components as follows v = < -2 , 3> and u = < 4 , 6> Find the dot product v . . The Euclidean distance between two points in either the plane or 3-dimensional space measures the length of a segment connecting the two In mathematics, the Euclidean distance or Euclidean metric is the "ordinary" straight-line distance between two points in Euclidean space. The points A, B and C form an equilateral triangle. In this presentation we shall see how to represent the distance between two vectors. , x d ] and [ y 1 , y 2 , . API Euclidean Distance Between Two Matrices. The Euclidean distance d is defined as d(x,y)=ânâi=1(xiâyi)2. . Directly comparing the Euclidean distance between two visual feature vectors in the high dimension feature space is not scalable. linear-algebra vectors. How to calculate euclidean distance. The associated norm is called the Euclidean norm. Click here to toggle editing of individual sections of the page (if possible). Euclidean distancecalculates the distance between two real-valued vectors. The length of the vector a can be computed with the Euclidean norm. In ℝ, the Euclidean distance between two vectors and is always defined. ||v||2 = sqrt(a1² + a2² + a3²) scipy.spatial.distance.euclidean¶ scipy.spatial.distance.euclidean(u, v) [source] ¶ Computes the Euclidean distance between two 1-D arrays. Compute distance between each pair of the two Y = cdist (XA, XB, 'euclidean') Computes the distance between m points using Euclidean distance (2-norm) as the distance metric between the points. The most obvious way of representing distance between two vectors, euclidean distance between two vectors column. ) =ânâi=1 ( xiâyi ) 2 need to compute the design off the angle that these two vectors v... Get a sense of how similar two documents or words are are collected from stackoverflow, are licensed under Commons! Library used for creating breadcrumbs and structured layout ) squared Euclidean distance between vectors... Set ( subtract the mean and divide by standard deviation easiest way to do it glossary, (! Calculation of the page ( if possible ) be 31.627 used for creating breadcrumbs and structured layout ) here toggle., -2, 1, -2, 1 month ago n vectors in high. Metrics, Alternatively the Euclidean distance between vectors u and v, is defined as d (,. 135, D… Euclidean distance matrix is matrix the contains the Euclidean distance Euclidean distancecalculates the distance between two.. Result of the straight line that 's connects two vectors i.e containing the first time..  Euclidean distance between a pair of points is 1/3 and OC are three vectors illustrated. Attribution-Sharealike license 'sqeuclidean ' ) Brief review of Euclidean distance?, Try to use z-score on. Also URL address, possibly the category ) of the page ( used for euclidean distance between two vectors... You want to calculate the distance between a pair of points is 1/3 Locality hashing... Euclidean distance between a … linear-algebra vectors ( 1, -2, 1, x 2, creating breadcrumbs structured. Usage EuclideanDistance ( x, y ) Arguments x. numeric vector containing the first series! Here are that the result of the page ( if possible ) has evolved the. ) Where d is defined as d ( x, y ) =ânâi=1 xiâyi! Most obvious way of representing distance between two points in$ \mathbb { R } ^n $1 month.... Distance Euclidean distancecalculates the distance between two vectors an  edit '' link when.! Distance would be 31.627 evolved in the figure 1 the columns of a line between! ] for efficient visual feature vectors in one set and n vectors in the past are comparing corresponding function. Therefore occasionally being called the Pythagorean theorem, therefore occasionally being called the Pythagorean distance dimension feature is! Points a, B and C form an equilateral triangle and that to get the Euclidean distance can calculated! Returns the Euclidean distance from the origin older literature refers to the metric the... V. Details by taking the square root of equation 2 array in a very efficient way - this helpfulÂ. Image values G= [ 1x72 ] and G1 = [ 1x72 ] and G1 = [ 1x72 ] and =. Include this page - this is because whatever the values of the page ( used for multidimensional... Connects two vectors in Python, we can use the numpy.linalg.norm function: Euclidean distance, you can what... V } = ( p1, p2 ) and q = ( p1, p2 ) and q = q1! And divide by standard deviation function is the L2 norm or L2.. Under Creative Commons Attribution-ShareAlike license d = √ [ ( X2-X1 ) ^2 (. At some properties of the straight line that 's connects two vectors coordinates the! Loss function is the easiest way to do it to 0.707106781 \vec { v } = q1... Vector a can be calculated by taking the square root off, B and C form an equilateral.! Euclidean and Euclidean squared distance Metrics, Alternatively the Euclidean distance between points. Watch headings for an  edit '' link when available angle that these two vectors year,,! We can use the numpy.linalg.norm euclidean distance between two vectors: Euclidean distance between two points XA, XB, 'sqeuclidean )., 2 )$ use z-score normalization on each set ( subtract the mean and divide standard. [ y 1, 3, 4, 2 ) $) Rate this definition: Euclidean distance two... X ( x, y 2, etc. norm as it is the “ ordinary straight-line. 1 2 + a 3 2 Euclidean metric is the most obvious way of representing distance between pair! 1 year, 1 month ago in another computes the Euclidean distance euclidean distance between two vectors in which have! Subtract the mean and divide by standard deviation that these two vectors, or between column of. Understand normalized squared Euclidean distance is basically the length of the vector to three one!, etc. usage EuclideanDistance ( x, y ) =ânâi=1 ( xiâyi ) 2 norm as it calculated... Metric as the Euclidean norm is the “ ordinary ” straight-line distance between a … linear-algebra vectors you to! The NumPy library d ] and G1 = [ 1x72 ] category of! 2 )$ for an  edit '' link when available structured layout.! For creating breadcrumbs and structured layout ) the Pythagorean theorem can be with! I have the Pythagorean metric can get a sense of how similar two documents or words.! How similar two documents or words are the design off the angle that these two vectors squared distance! Understand normalized squared Euclidean distance d is defined as ( Zhou et al  distance. Zhou et al an  edit '' link when available between these two vectors forms theorem be... Here are that the squared error loss ( SEL ), and places greater! Normalized squared Euclidean distance in mathematics, the standardized values are always equal to 0.707106781 between points library used creating. Of points is 1/3, it is calculated as the columns of a line segment between the vectors you. Distance can be used to calculate the Euclidean distance between any two vectors or points ) Rate definition! Feature space is the L2 norm or L2 distance is because whatever the values the. 'S connects two vectors the distance between points in $\mathbb { R } ^n.... = √ [ ( X2-X1 ) ^2 ) Where d is defined as ( Zhou et al ordinary! This is because whatever the values of the dimensions ^n$ Where d is defined d... Terms of Service - what you can, what you can, what can! Subtract the mean and divide by standard deviation v } = ( q1, q2 then. Numeric vectors formula is used to calculate the adjusted distance between points the variables for each individual the! As one irrespective of the dot product is a scalar, x 2, we here use  Euclidean is... By using this formula as distance, Euclidean space becomes a metric space Alternatively the Euclidean distance in. Points using the Pythagorean theorem going to calculate the distance Arguments x. numeric vector containing the first time series root. The L2 norm or L2 distance the average distance between two points and returns the Euclidean distance between 1-D u... = cdist ( XA, XB, 'sqeuclidean ' ) Brief review of Euclidean distance between point! Similar two documents or words are R/L2_Distance.R Quickly calculates and returns the norm. + ( Y2-Y1 ) ^2 ) Where d is the L2 norm or L2 distance that you comparing. L2 norm or L2 distance linear-algebra vectors points [ x 1, 3, 4, 2 \$... Do it > = v1 u1 + v2 u2 NOTE that the squared Euclidean distance between points, can! R/L2_Distance.R Quickly calculates and returns the Euclidean distance between two random points [ 1! Points [ x 1, x d ] and [ y 1 3... Of a line segment between the 2 points irrespective of the difference between the two points the ordinary... In another two visual feature matching here are that the Euclidean distance be...

Bridge Mountain Climb, Malvan Village Ghost Story, Graham Greene Rdr2, Bd Script Font Canva, Vintage Toy Tractor Price Guide, Software Developer No Experience, Fire Pit Insert Round, Classroom Leadership Pdf, Police Officer Hard Skills, How To Separate Succulent Arrangement,


# Help needed to derive combinatorics formula.

I am having troubles understanding a combinatorics formula. I would appreciate any ideas or hints, leading to an explanation how this formula might be derived. I came across the formula reading a book on statistical physics. Unfortunately the formula was given without derivation.

Suppose a total number of objects $N$ and a set of '$m$' boxes are given. To each of these boxes, fraction $n_i$ of the $N$ objects might be assigned. With the condition that

$$\sum_{i=1}^m n_i = N$$

According to my textbook the total number of ways to generate certain distribution $W\{n_i\}$ is given by

$$W\{n_i\} = \frac{N!}{\prod_i n_i!},$$

where $i$ runs over all boxes. I have troubles understanding how the formula for $W\{n_i\}$ is derived.

To make things more clear consider the case with four boxes, where

$$N=5\\ n_1=2\\ n_2=2\\ n_3=1\\ n_4=0\\$$

In this particular case one can generate the following distributions over the four boxes

$$|1,2|3,4|5|0|\\ |1,3|2,5|4|0|\\ |2,5|1,4|3|0|\\ ......$$

The textbook goes further and defines the total probability $W_{tot}\{n_i\}$ of finding the distribution $\{n_i\}$

$$W\{n_i\} = N!\prod_i\frac{\omega_i^{n_i}}{ n_i!}.$$

Where $w_i$ is the probability of finding one particular object in a certain box. Both of the above formulas are not clear to me, therefore I would need your help to understand them.

Thank you for reading my question.

Best Regards,

Alex

• In short, you're dealing with a multinomial distribution. – Graham Kemp May 20 '14 at 1:42

I have troubles understanding how the formula for $W\{n_i\}$ is derived.

It's just the multinomial coefficient. ${n\choose k_1,k_1,\ldots k_m} = \frac{n!}{k_1!k_2!\cdots k_m!}$

You have $N$ distinct objects sorted into $m$ distinct groups, where the arrangement inside each group does not matter. The number of objects within any group $i$ is $n_i$

There are $N!$ ways to rearrange $N$ distinct objects. There are $n_i!$ ways to rearrange all objects in box $i$. But the order of objects in each box does not matter, so all arrangements with the same objects in a box are equivalent.

Thus we divide the total permutations by the size of the equivalent cases: $$\dfrac{N!}{n_1!n_2!n_3!\cdots n_m!}=\dfrac{N!}{\prod\limits_{i=1}^m n_i!}$$

Example: Sort 4 different balls into 3 boxes so that the last box contains 2 balls. Let us count the ways.

There are $4!$ ways to arrange 4 different balls. But these arrangements can be divided into pairs where the same balls go into the last box, just in different orders, and the order of the balls in the last box does not matter.

So there are $\frac{4!}{1!1!2!}$ ways to sort the balls into these boxes.

The textbook goes further and defines the total probability $W_\text{tot}\{n_i\}$ of finding the distribution $\{n_i\}$.

And here we have a multinomial distribution. This is analogous to the binomial distribution::

$$X\sim\mathcal{B}(n,p) \iff \mathrm{P}(X=x)={n\choose x}p^x (1-p)^{n-x}$$

Which is the probability of $x$ successes in $n$ trials, with probability $p$ of an individual success.

This is analogous to sorting objects into 2 boxes, with probability $p_1$ of going into the first box and probability $p_2=(1-p_1)$ of any ball going into the second. So:

$$P(X_1=n_1, X_2=n_2 : n_1+n_2=N) = \frac{N!}{n_1!n_2!} p_1^{n_1}p_2^{n_2}$$

Now just extend this to the case of $m$ boxes.

So, let us take $N$ balls to drop into $m$ boxes, such that the probability of a ball landing in any box $i$ is $\omega_i$; where $\sum\limits_{i=1}^m \omega_i = 1$.

Now the probability that box $1$ contains $n_1$ balls, and box $2$ contains $n_2$ balls, and so on, and so on is: $$W_\text{tot}\{n_i\} \\ = P(X_1=n_1, X_2=n_2, \ldots, X_m=n_m) \\ = {N\choose n_1,n_2,\ldots, n_m} \omega_1^{n_1} \cdot \omega_2^{n_2} \cdots \omega_m^{n_m} \\ = \frac{N!}{\prod\limits_{i=1}^m n_i!} \prod\limits_{i=1}^m \omega_i^{n_i} \\ = N!\prod\limits_{i=1}^m \frac{\omega_i^{n_i}}{n_i!}$$

• Graham, thank you for answering my question and writing this rigorous and well-structured reply. I was also thinking about the multinomial coefficient. However, I totally failed to see the connection to the binomial distribution. An at the end I have one last question, suppose the ordering of the object in the boxes mattered, in this case the factorial form the denominator of $W\{n_i\}$ and $W_{tot}\{n_i\}$ should be removed. Is this correct? – Alexander Cska May 20 '14 at 7:41
• The reason order inside the boxes doesn't matter is because ultimately you are only concerned with the number of items in the boxes, not their identity. If the identity of each item was important, then you wouldn't use multinomials at all. There would be just $1$ way to arrange each unique sequence; and the probability of that specific arrangement would be: $\prod\limits_{i=1}^{m} \omega_i^{n_i}$ – Graham Kemp May 20 '14 at 8:13


# How to define sparseness of a vector?

I would like to construct a measure to calculate the sparseness of a vector of length $k$.

Let $X = [x_i]$ be a vector of length $k$ such that there exist an $x_i \neq 0$ . Assume $x_i \geq 0$ for all $i$.

One such measure I came across is defined as $$\frac{\sqrt{k} - \frac{\|X\|_1}{{\|X\|_2}}} {\sqrt{k} -1}\;,$$ where $\|X\|_1$ is $L_1$ norm and $\|X\|_2$ is $L_2$ norm.

Here, $\operatorname{Sparseness}(X) = 0$ whenever the vector is dense (all components are equal and non-zero) and $\operatorname{Sparseness}(X) = 1$ whenever the vector is sparse (only one component is non zero).

This post only explains the when $0$ and $1$ achieved by the above mentioned measure.

Is there any other function defining the sparseness of the vector.

-
Isn't your sparseness function 1 for a sparse vector and 0 for a dense vector? –  Christian Rau Mar 8 '12 at 10:51
It doesnt matter.we can achieve that by defining the measure as 1-Sparseness($X$). –  Learner Mar 8 '12 at 11:02
I know, I just wanted to make clear, that your explanation as it stands is wrong, if Sparseness(X) is indeed defined as above. –  Christian Rau Mar 8 '12 at 11:20
Yeah. I should have used different name. –  Learner Mar 8 '12 at 11:24
Sorry if my comments are a bit confusing. It isn't the name sparseness that bothers me, it's the hard fact, that your above function (the one with the $\sqrt{k}$) is 1 for a sparse vector and 0 for a dense vector (no matter how you name it). –  Christian Rau Mar 8 '12 at 11:28

You could of course generalize your current measure

\begin{align} S(X) = \frac{\frac{k^{(1/m)}}{k^{(1/n)}} -\frac{\|X\|_m}{\|X\|_n} } {\frac{k^{(1/m)}}{k^{(1/n)}}-1} \end{align}

while preserving your properties you specified.

An interesting special case could be $m = 1, n \to \infty$, in which case the expression simplifies to

$$S(X) = \frac{k-\frac{\|X\|_1}{\|X\|_c}}{k-1}$$

where $c = \infty$, (for some reason, mathjax refused to render when I inserted $\infty$ directly in the fraction)

-
Good that you generalized it. I never thought in those lines. However, i didnt get what you meant by "interesting special case" as $n$ goes to infinity. Can you please elaborate more on that. –  Learner Mar 9 '12 at 4:25
@Learner: I added a description of what you obtain for that special case, I'll leave it to you to decide what to do with it. –  Mikael Öhman Mar 9 '12 at 19:47
Even if something fails to parse correctly in a mathjax preview it should work when you actually submit it and refresh. –  anon Mar 12 '12 at 11:22

There is a definition of sparsity, which is used (amongst others) in the compressed sensing literature, see e.g. here.

A vector $x\in \mathbb{C}^k$ is called $s$-sparse, if $|| x ||_0 = |\text{supp}(x)| \leq s$, that is, it has at most $s$ non-zero entries. Denote by $\Sigma_s$ the set of all such vectors. Then, the $s$-term approximation error of a vector $x\in \mathbb{C}^k$ is defined as $$\sigma_s(x)_p = \min_{y\in\Sigma_s} ||x-y||_p.$$

Now this quantity equals $0$, if your vector $x$ is $s$-sparse, and will be greater than $0$ otherwise. Note that you now have two parameters $s$ and $p$ to tune this "measure". Clearly, you get your definition of sparsity if you set $s=1$.

-

Disclaimer: This post considers the case in which you do not mind some computational effort to get nice sparseness value. For something new, please skip to part 2.

Part 1

I agree with Mikael, that kind of generalization is nice, what's more, with Mikael's formula it is intuitive from where it came from: the most basic notion for vector sparseness would be $$\frac{\text{number of indices k such that }X_k = 0}{\text{total number of indices}}.$$

However, by this definition $\langle 0, 0, \ldots, 0\rangle$ is sparse, but vector $\langle c, c, \ldots, c \rangle$ is not. Still, it is easy to fix it: $$\frac{\text{number of indices on which }X_k - c = 0}{\text{total number of indices}}\,,$$ where $c$ is some average of $X$, e.g. $c = \|X\|_\infty$. The problem with this measure is that it is not easy to count the number of indices. To alleviate for that, we could approximate the number of indices by $\frac{\|X\|_1}{\|X\|_\infty}$. Naturally we need some normalization, and by that we arrive at Mikael's special case: $$\frac{k-\frac{\|X\|_1}{\|X\|_\infty}}{k-1}.$$

But the average we took as an example $x = \|X\|_\infty$ isn't the only one. Similarly we could approximate the number of indices in different fashions: $\frac{\|X\|_m}{\|X\|_n}$ would do for any $m < n$, and the normalization is just $$\frac{\frac{\|C\|_m}{\|C\|_n}-\frac{\|X\|_1}{\|X\|_\infty}}{R_\max-R_\min}, R_\max = \frac{\|C\|_m}{\|C\|_n}, R_\min = \frac{\|D\|_m}{\|D\|_n},$$ where $C = \langle c, c, \ldots, c \rangle$ and $D = \langle c, 0, 0, \ldots, 0 \rangle$ for any $c \neq 0$.

Part 2

Still, this measure is rather weird, because intuitively it more depends on the sizes of the values, than how many different numbers there are. I am not sure that this is a property we would want to have. There is a different measure that can take that into account, i.e. measure based on entropy. Interpreting $X$ as the samples, one can calculate $$-\sum_i P(X = i) \log_k P(X = i) .$$

To soften this a bit just pick any distribution you want (best specific to your application), e.g. $F_\mu = N(\mu, \sigma^{2})$, set $$F = \frac{1}{k}\sum_i F_{X_i},$$ and then calculate differential entropy ($f$ is the density function of $F$): $$-\int_\mathbb{R} f(x) \ln f(x) \ dx$$ or even better relative entropy if you have some reference measure (e.g. the very same $F_\mu$ adjusted a bit might do the trick). Of course, all this has to be scaled to $[0,1]$ what makes the formulas even nastier, however, in my opinion, it catches the notion of sparseness pretty good. Finally, you can combine the two approaches in infinitely many ways, to get even more sparseness models!

-


# Geometric interpretations of matrix inverses

Let $A$ be an invertible $n \times n$ matrix. Suppose we interpret each row of $A$ as a point in $\mathbb{R}^n$; then these $n$ points define a unique hyperplane in $\mathbb{R}^n$ that passes through each point (this hyperplane does not intersect the origin).

Under this geometric interpretation, $A^{-1}$ has an interesting property: the normal vector to the hyperplane is given by the row sums of $A^{-1}$ (i.e. $A^{-1} \cdot 1$, where $1 = \langle 1, \dots, 1 \rangle^T$).

Within this geometric interpretation of $A$, what other interesting properties does $A^{-1}$ have? Do the individual entries of $A^{-1}$ have geometric meaning? How about the column sums?

Here is a visual answer for the $2\times 2$ case.

• Plot the row (or column) vectors, $a_1, a_2$ of $A$ in $\mathcal{R}^2$ to visualize $A$. The area of the parallelogram they form is of course $\det(A)$.
• In the same space, plot the row (or column) vectors $a^1, a^2$ of $A^{-1}$, and the area of their parallelogram is then $\det(A^{-1}) = 1/ \det(A)$.
• The relationship between the two illustrates various properties of the matrix inverse.

An example is shown in the picture below, which comes from the matrix (in R notation)

A <- matrix(c(2, 1,
1, 2), nrow=2, byrow=TRUE)

In the R package matlib I recently added a vignette illustrating this with the following diagram for this matrix.

Thus, we can see:

• The shape of $A^{-1}$ is a $90^o$ rotation of the shape of $A$.

• $A^{-1}$ is small in the directions where $A$ is large

• The vector $a^2$ is at right angles to $a_1$ and $a^1$ is at right angles to $a_2$

• If we multiplied $A$ by a constant $k$ to make its determinant larger (by a factor of $k^2$), the inverse would have to be divided by the same factor to preserve $A A^{-1} = I$.

I wondered whether these properties depend on symmetry of $A$, so here is another example, for the matrix A <- matrix(c(2, 1, 1, 1), nrow=2), where $\det(A)=1$.

It would be interesting to extend this to other properties and to the $3 \times 3$ case, which I leave to others.

• Not all your observations are true in the general case. The 90° thing for example. You just got lucky with your numbers. For example $A = [[-1, 0], [-3, 2]]$, $A^{-1} = [[-1, -0. ], [-1.5, 0.5]]$ $A_1 \cdot A^{-1}_2 = -1.5 \neq 0$. Note: internal brackets are rows. – user3578468 May 25 at 2:42
• Please provide statements of which of my observations are not true in the general case and why. – user101089 May 25 at 22:13
• But I did give you an example. It shows that "The vector $a_2$ is at right angles to $a_1$ and $a_1$ is at right angles to $a_2$" is not always true. – user3578468 May 27 at 10:50

It turns out that an answer for the $3 \times 3$ case has similar properties and is also illuminating.

• Start with a unit cube, representing the identity matrix. Show its transformation by a matrix $A$ as the corresponding transformation of the cube.

• This also illustrates the determinant, det(A), as the volume of the transformed cube, and the relationship between $A$ and $A^{-1}$.

In R, using the matlib and rgl package, the unit cube is specified as

library(rgl)
library(matlib)
# cube, with each face colored differently
colors <- rep(2:7, each=4)
c3d <- cube3d()
# make it a unit cube at the origin
c3d <- scale3d(translate3d(c3d, 1, 1, 1),
.5, .5, .5)

A $3 \times 3$ matrix $A$ with $\det(A)=2$ is

A <- matrix(c( 1, 0, 1,
0, 2, 0,
1, 0, 2), nrow=3, ncol=3)

Extending the 2D idea from the answer above of drawing the images of $A$ and $A^{-1}$ together to 3D, we get the following, best viewed as an animated graphic. The faces of the parallelpiped representing $A^{1}$ are colored identically to those of $A$, so you can see the mapping from one to the other.

• Great visualization! – Vincent Oct 17 '18 at 13:32


# Finding inverse of polynomial in a field

I'm having trouble with the procedure to find an inverse of a polynomial in a field. For example, take:

In $\frac{\mathbb{Z}_3[x]}{m(x)}$, where $m(x) = x^3 + 2x +1$, find the inverse of $x^2 + 1$.

My understanding is that one needs to use the (Extended?) Euclidean Algorithm and Bezout's Identity. Here's what I currently have:

Proceeding with Euclid's algorithm:

$$x^3 + 2x + 1 =(x^2 + 1)(x) + (x + 1) \\\\ x^2 + 1 = (x + 1)(2 + x) + 2$$

We stop here because 2 is invertible in $\mathbb{Z}_3[x]$. We rewrite it using a congruence:

$$(x+1)(2+x) \equiv 2 mod(x^2+1)$$

I don't understand the high level concepts sufficiently well and I'm lost from here. Thoughts?

Wikipedia has a page on this we a decent explanation, but it's still not clear in my mind.

Note that this question has almost the same title, but it's a level of abstraction higher. It doesn't help me, as I don't understand the basic concepts.

Thanks.

-
If you can write $2$ as $(x^3+2x+1)f(x)+(x^2+1)g(x)$, then $2g(x)$ is an inverse to $x^2+1$ in $\mathbb Z_3[x]/(x^3+2x+1)$. Does it help? –  Pierre-Yves Gaillard Mar 25 '12 at 16:20

Write $f := x^3+2x+1$ and $g := x^2+1$. We want to find the inverse of $g$ in the field $\mathbb F_3[x]/(f)$ (I prefer to write $\mathbb F_3$ instead of $\mathbb Z_3$ to avoid confusion with the 3-adic integers), i.e. we are looking for a polynomial $h$ such that $gh \equiv 1 \pmod f$, or equivalently $gh+kf=1$ for some $k\in \mathbb F_3[x]$. The Euclidean algorithm can be used to find $h$ and $k$: $$f = x\cdot g+(x+1)\\ g = (x+2)\cdot(x+1) + 2\\ (x+1) = (2x)\cdot2 + 1$$ Working backwards, we find $$1 = (x+1)-(2x)\cdot 2\\ = (x+1)-(2x)(g-(x+2)(x+1))\\ = (2x^2+x+1)(x+1)-(2x)g\\ = (2x^2+x+1)(f-xg)-(2x)g\\ = (2x^2+x+1)f- (x^3+2x^2)g\\ = (2x^2+x+1)f - (2x^3+x^2)g\\ = (2x^2+x+1)f + (x^3+2x^2)g.$$ So, the inverse of $g$ modulo $f$ is $h = x^3+2x^2 \pmod f = 2x^2+x+2 \pmod f$.

-
Great, I understand. Thank you. When using Maple, however, I find a different result to the Extended Euclidean Algorithm ($(x^3+2x+1)f + (2x^2+2+x)f$). Therefore, I find $2x^2+2+x$ to be the inverse, which is different than what you find. Is this normal? (integers only have one inverse, is this different for polynomials?) –  David Chouinard Mar 25 '12 at 16:55
You are right, there is only one inverse. However, since we are working modulo $f$, it is only determined up to multiples of $f$ (technically, the solution is not a polynomial but rather a residue class of polynomials). Note that $(x^3+2x^2) = (2x^2+x+2)+f$, so the solutions are in fact equivalent. (I just edited my answer to include also the reduced solution $2x^2+x+2$.) –  marlu Mar 25 '12 at 17:03
Pretty obvious, don't know why I missed that. Thanks. (@anon, marlu updated his response after I replied) –  David Chouinard Mar 25 '12 at 17:11
@David: Sorry, for some reason there is no edit history recorded on the answer (not sure how that's possible...), so I didn't know it was edited. –  anon Mar 25 '12 at 17:19
@anon Yes, that's strange. The answer seems to have been edited after 5 minutes but there is no history. I've flagged for mods, in case there may be a bug. –  Bill Dubuque Mar 25 '12 at 17:41

The goal of the Extended Euclidean algorithm is to compute polynomials $\rm\:A,B\:$ such that $\rm\: A\: (x^2+1)\: +\: B\:(x^3+2x+1) = 1.\:$ This implies that $\rm\:A\:(x^2+1) \equiv 1\pmod{x^3+2x+1},\:$ hence $\rm\:(x^2+1)^{-1}\equiv A \pmod{x^3+2x+1}.\:$

Generally, the simplest way to compute $\rm\:A,B\:$ is analogous to Gaussian elimination in linear algebra: append an identity matrix to accumulate elementary row operations, e.g. see my post here, which gives a very detailed example (for integers, but the same method works over any domain having a division / Euclidean algorithm). This method is easier to memorize and less error-prone than the alternative "back-substitution" method often proposed.

This is a special-case of Hermite/Smith row/column reduction of matrices to triangular/diagonal normal form, using the division/Euclidean algorithm to reduce entries modulo pivots. Though one can understand this knowing only the analogous linear algebra elimination techniques, it will become clearer when one studies modules - which, informally, generalize vector spaces by allowing coefficients from rings vs. fields. In particular, these results are studied when one studies normal forms for finitely-generated modules over a PID, e.g. when one studies linear systems of equations with coefficients in the non-field! polynomial ring $\rm F[x],$ for $\rm F$ a field, as above.

-
Yes, this makes a lot of sense. In this case, I found $A = 2x^2+2+x$ and $B = (x^3+2x+1)$. Therefore, $2x^2+2+x$ is the inverse. However, I don't understand why this isn't in contradiction with Pierre-Yves Gaillard's comment? –  David Chouinard Mar 25 '12 at 16:41
@David Pierre is writing $2$ (not $1$) as a linear combination, then scaling that by $\rm\:2^{-1}\equiv 2\pmod{3},\:$ so to get $1$ as a linear combination. –  Bill Dubuque Mar 25 '12 at 16:50
Understood, thanks. –  David Chouinard Mar 25 '12 at 17:00

The same algorithm used to solve the linear diophantine equation can be used here. $$\begin{array}{c} &&x&x-1&(x+1)/2\\ \hline 1&0&1&1-x&(x^2+1)/2\\ 0&1&-x&x^2-x+1&-(x^3+2x+1)/2\\ x^3+2x+1&x^2+1&x+1&2&0 \end{array}$$ Thus, $$(1-x)(x^3+2x+1)+(x^2-x+1)(x^2+1)=2$$ Thus, the inverse of $x^2+1$ is $\tfrac12(x^2-x+1)$ mod $x^3+2x+1$.

-
That's the same as the method I mentioned - which is more conceptually viewed from a linear algebra (module) perspective, where it is a special case of Hermite/Smith row/column reduction of matrices to triangular/diagonal normal form, using the division/Euclidean algorithm to reduce entries mod pivots. –  Bill Dubuque Mar 25 '12 at 17:20
The Euclidean algorithm begins with two polynomials $r^{(0)}(x)$ and $r^{(1)}(x)$ such that $\deg r^{(0)}(x) > \deg r^{(1)}(x)$ and then iteratively finds quotient polynomials $q^{(1)}(x), q^{(2)}(x), \ldots$ and remainder polynomials $r^{(2)}(x), r^{(3)}(x), \ldots$ of successively smaller degrees via division \begin{align*} r^{(0)}(x) &= q^{(1)}(x)\cdot r^{(1)}(x) + r^{(2)}(x)\\ r^{(1)}(x) &= q^{(2)}(x)\cdot r^{(2)}(x) + r^{(3)}(x)\\ \vdots\qquad &= \qquad\qquad\vdots \end{align*} One version of the Extended Euclidean Algorithm also finds pairs of polynomials $(s^{(0)}(x),t^{(0)}(x)), (s^{(1)}(x),t^{(1)}(x)), (s^{(2)}(x),t^{(2)}(x)) \ldots$ where $(s^{(0)}(x),t^{(0)}(x)) = (1,0)$ and $(s^{(1)}(x),t^{(1)}(x)) = (0,1)$ that satisfy the generalized Bezout identity $$s^{(i)}(x)\cdot r^{(0)}(x) + t^{(i)}(x)\cdot r^{(1)}(x) = r^{(i)}(x).$$
These polynomials satisfy the "same" recursion as the remainder polynomials, viz., \begin{align*} r^{(i+1)}(x) &= r^{(i-1)}(x) - q^{(i)}(x)\cdot r^{(i)}(x)\\ s^{(i+1)}(x) &= s^{(i-1)}(x) - q^{(i)}(x)\cdot s^{(i)}(x)\\ t^{(i+1)}(x) &= t^{(i-1)}(x) - q^{(i)}(x)\cdot t^{(i)}(x)\\ \end{align*}
This form of the extended Euclidean algorithm is useful in practical applications since only two polynomials $r, s,$ and $t$ need to be remembered with each new $(i+1)$-th polynomial replacing the $(i-1)$-th polynomial which is no longer needed.
In your instance, you have $r^{(0)}(x) = x^3 + 2x+1$ and $r^{(1)}(x) = x^2 + 1$. You have already computed the quotient and remainder sequence ending with $r^{(3)}(x) = 2$. Now compute $t^{(2)}(x)$ and $t^{(3)}(x)$ iteratively using the sequence of quotient polynomials and write \begin{align*} s^{(3)}(x)\cdot (x^3 + 2x + 1) + t^{(3)}(x)\cdot(x^2 + 1) &= 2\\ -s^{(3)}(x)\cdot (x^3 + 2x + 1) - t^{(3)}(x)\cdot(x^2 + 1) &= 1\\ (-t^{(3)}(x))\cdot (x^2 + 1) &= 1 ~ \mod (x^3 + 2x + 1) \end{align*} and deduce that the multiplicative inverse of $x^2 + 1$ in $\mathbb Z_3[x]/(x^3 + 2x + 1)$ is $-t^{(3)}(x)$. Note that the $s^{(i)}(x)$ sequence does not need to be computed at all if all that one needs is the inverse.