# Which payoff do you want to go for?

Payoff 1.

Toss 5 coins. You get $1 for each consecutive pair HT that you get. Payoff 2. Toss 5 coins. You get$1 for each consecutive pair HH that you get.

The consecutive pairs are allowed to overlap.

For example, if you tossed HTHHH, under payoff 1 you will get $1, but under payoff 2 you will get$2.

Note by Calvin Lin
4 years, 7 months ago

This discussion board is a place to discuss our Daily Challenges and the math and science related to those challenges. Explanations are more than just a solution — they should explain the steps and thinking strategies that you used to obtain the solution. Comments should further the discussion of math and science.

When posting on Brilliant:

• Use the emojis to react to an explanation, whether you're congratulating a job well done , or just really confused .
• Ask specific questions about the challenge or the steps in somebody's explanation. Well-posed questions can add a lot to the discussion, but posting "I don't understand!" doesn't help anyone.
• Try to contribute something new to the discussion, whether it is an extension, generalization or other idea related to the challenge.

MarkdownAppears as
*italics* or _italics_ italics
**bold** or __bold__ bold
- bulleted- list
• bulleted
• list
1. numbered2. list
1. numbered
2. list
Note: you must add a full line of space before and after lists for them to show up correctly
paragraph 1paragraph 2

paragraph 1

paragraph 2

[example link](https://brilliant.org)example link
> This is a quote
This is a quote
    # I indented these lines
# 4 spaces, and now they show
# up as a code block.

print "hello world"
# I indented these lines
# 4 spaces, and now they show
# up as a code block.

print "hello world"
MathAppears as
Remember to wrap math in $$ ... $$ or $ ... $ to ensure proper formatting.
2 \times 3 $2 \times 3$
2^{34} $2^{34}$
a_{i-1} $a_{i-1}$
\frac{2}{3} $\frac{2}{3}$
\sqrt{2} $\sqrt{2}$
\sum_{i=1}^3 $\sum_{i=1}^3$
\sin \theta $\sin \theta$
\boxed{123} $\boxed{123}$

Sort by:

This is a trick question, both payoffs have same expected value of profit, which is: $1. Consider this: We will count the number of ways HT can occur. We can have: HT_ _ _ : 8 ways _ HT _ _ _ : 8 ways _ _ HT _ : 8 ways _ _ _ HT : 8 ways Total number of ways: 32 Similarly for HH, the total number of ways in which it can occur is 32. What this means is that the sum of the payoffs of HT and HH over the 32 possible outcomes is$32 for both. The payoffs may be distributed in different ways for HT and HH, but their expected value is same.

Mathematically, the expected value is same. But there is one more thing left to do: analyze the probability distribution. We can easily see that payoff 2 might be desirable because it tends to give large profits as compared to payoff 1 for certain events. eg: HHHHH gives $4 in p2 and 0 in p1. HHHHT gives$3 in p2 and $1 in p1. But the thing is that these events are not very common. Using method of reflection and analyzing the 16 possibilities , we see that there are many events for which p2 gives$0 payoff. There are 13 outcomes for which p2 gives $0 payoff, but only 6 where p1 gives$0 payoff.

So If I'm given this choice, I'd take payoff 1, as then I have more chance of leaving with at least a dollar in my pocket.

Since this is in quantitative finance, should I talk about risk and stuff? I do not know.

- 4 years, 7 months ago

I think that there may be some double-counting here. For example, you've counted HTHTT and HTHTH twice each. There are only $2^{5} = 32$ possible sequences in total, and there are some, such as TTHHH, that have no HT payouts, so there cannot be $32$ HT outcomes. I'm getting an average payout of $\dfrac{28}{32}$ dollars in the first game type and $\dfrac{31}{32}$ dollars in the second, but I am too tired to double-check. I'll do that tomorrow. :)

Edit: Oh. now I see what you've done; very clever. I must have missed out on my count somewhere. I'll still wait to confirm in the morning. And yes, there does seem to be an advantage to choosing p1, (even though the expected winnings are the same), in the sense that you are more likely to at least win some money.

- 4 years, 7 months ago

Is "more likely to at least win some money" the main consideration that you will use if the expected payoff is equal?

For example,what would you choose between:

Payoff 1: 50% of $0, 50% of$100
Payoff 2: 90% of $1.01, 1% of$4900

Staff - 4 years, 7 months ago

If Payoff 2 was 99% of $40.40, 1% of$1000, (to give it the same expected payoff as Payoff 1), I would go with Payoff 2 since I would be assured of money and would at least have a slim chance of a large amount of money.

If Payoff 2 was 99% of $0, 1% of$5000, then it gets interesting, (at least for me). Even a such a slight chance of winning $5000 would be hard to pass up, so I would probably go with Payoff 2, unless I absolutely needed that$100 right away.

If Payoff 2 was 99.9% of 0$, 0.1% of$50000 .... hmmmm .... That's a lot of dough, so I'd go with Payoff 2. However, if the options were these last two I've listed, then I might choose the $5000 option. If Payoff 2 involved a 50% chance of losing$100 and a 50% chance of winning $200, then I would go with Payoff 1, since I'm not comfortable with there being such a good chance of losing a fair bit of money. There is a lot of psychology going on in these choices, and yet the expected winnings is always$50, (which ironically is an amount that would never actually be won in any of these scenarios).

- 4 years, 7 months ago

(Ooops, apparently I can't do maths. Fixed.)

So, there are multiple ideas here.
- Is the probability of a positive payoff so important that just because it moves from $0 it would greatly influence your preferences? - At what point does the potential promise of a huge payoff overweigh the certainty of a small payoff? Staff - 4 years, 7 months ago Log in to reply In general I guess it all depends on the specific values, and further, on each person's financial status and willingness to take on risk. For me, the potential, (1% or better), of a large payout would outweigh any guaranteed amount less than$100, so in this case there is no difference to me between $0 and$100. But if the guaranteed amount were, say, $5000, with a 0.01% all-or-nothing chance of winning$1,000,000, I'd probably take the $5000 and run, (although I would be tempted to take the risk, at least for a moment). If the all-or-noting percentage were 1%, though, then I would find it hard not to take the chance;$5000 is a lot of money, but $1,000,000 is a life-changing amount of money, and a 1% chance is realistic in my eyes given the potential reward. The question I would ask myself is: which will I regret more - not taking the guaranteed money or not taking the risk? - 4 years, 7 months ago Log in to reply Yes, the expected payoff for both scenarios is$1. The easiest way to see this is to look at Indicator Variables, which is essentially what Raghav did (though not phrased in that language).

Well, everyone has their own "payoff preference". For example, you stated that you would prefer to "be more likely to leave with at least a dollar in your pocket". As such, this tells me that you are very risk adverse, and that you would prefer the certainty of a positive payoff.

If you want to talk about "risk and stuff", what do you need to consider, and why?

Staff - 4 years, 7 months ago

I am not formally educated in these topics, so I'm just saying all this from a logical standpoint.

After a bit of thinking, I think I agree with you on the fact that everyone has their own "payoff preference". One can take a risk to win more, or take less risk to win less. In the scenario mentioned here, the aspects of risk are not conspicuous. The amount to be won is not significant, and there is no penalty for losing. Hence, both the payoffs are inherently attractive offers. When we come to real life situations, I think there are many more things that contribute to the risk:

1. The probability that you will get a payoff.

2. The probability that you stand to lose money.

3. The security of the winnings. Money once won shouldn't be taken away from you.

4. The effects of said decisions on long term/ on your ability to take other decisions.

According to me, every decision of ours is based on weighing in these risk factors against the probable prize. The balance may tip either way, and so may our decisions.

I don't even know if I'm thinking in the right direction... Help me out here @Calvin Lin

- 4 years, 7 months ago

Part of this series of posts is for you to figure out what kind of risk appetite you have. It is a personal preference, and there isn't necessarily a "right" answer. Your understanding will help guide you in decisions that you make in future. For the points that you raised, you should answer for yourself if those should matter, and why.

Hopefully, you will come up with a consistent, logical risk-reward structure. For example, if you would choose payoff 1 over 2 and payoff 3 over 4, but would prefer a combined 2 and 4 over a combined 1 and 3, then it would be very easy for someone to sell you things in part, and make you overpay for them.

Staff - 4 years, 7 months ago

@Calvin Lin sir, Am I right?

- 4 years, 7 months ago

Let $X$ be an indicator variable which takes on value 1 for each $HT$ pair and 0 otherwise, and let $Y$ be an indicator variable which takes on value 1 for each $HH$ pair and 0 otherwise. There are $5 - 2 + 1 = 4$ indicator variables (both $X$ and $Y$) in 5 coin tosses. The expected value of the payoff is: $\text{E}\left [\sum_{k = 1}^{4} X_{k} \right ], ~ \text{E}\left [\sum_{k = 1}^{4} Y_{k} \right ].$We can apply Linearity of Expectation here, which holds even for dependent events (nice proof of this is given in Brilliant Wiki Page), to get: $\sum_{k = 1}^{4}\text{E}\left [ X_{k} \right ] , ~ \sum_{k = 1}^{4}\text{E}\left [ Y_{k} \right ].$Expected value of both $X_{k}$ and $Y_{k}$ is: $\text{E}\left [ X_{k} \right ] = \text{E}\left [ Y_{k} \right ] = 1 \cdot \dfrac{1}{4} + 0 \cdot \dfrac{3}{4} = \dfrac{1}{4}.$It follows that expected payoffs in both cases are the same and they equal:$\sum_{k = 1}^{4}\text{E}\left [ X_{k} \right ] = \sum_{k = 1}^{4}\text{E}\left [ Y_{k} \right ] = 4 \cdot \dfrac{1}{4} = 1.$

Bonus: What about the variance of the payoff? It can also influence our decision. Is it also the same?

It turns out they are not! Let us first consider payoff 1 and indicator variable $X$:\begin{aligned} \text{Var}[X] &= \text{E}\left [\left(\sum_{k = 1}^{4}X_{k}\right)^{2}\right ] - \text{E}\left [\sum_{k = 1}^{4}X_{k}\right ]^{2} \\ &= \text{E}\left [\left(\sum_{k = 1}^{4}X_{k}^{2}\right) + \left(\sum_{k \neq j}X_{k}X_{j}\right)\right ] - \text{E}\left [\sum_{k = 1}^{4}X_{k}\right ]^{2} \\ &= \text{E}\left [\left(\sum_{k = 1}^{4}X_{k}^{2}\right)\right] + \text{E}\left[ \left(\sum_{k \neq j}X_{k}X_{j}\right)\right ] - \text{E}\left [\sum_{k = 1}^{4}X_{k}\right ]^{2} \\ &= 1 + \text{E}\left[ \left(\sum_{k \neq j}X_{k}X_{j}\right)\right ] - 1 \\ &= \text{E}\left[ \left(\sum_{k \neq j}X_{k}X_{j}\right)\right ].\end{aligned}The same is true for payoff 2 and $Y$. Now, let us calculate $\text{E}\left[ \left(\sum_{k \neq j}X_{k}X_{j}\right)\right ]$. We make two groups of $X_{k}X_{j}$:

• $\left | k-j \right | = 1$ representing two consecutive indicator variables which share one common coin. Notice that their product is always 0 since it's impossible to have two $HT$s in 3 consecutive coins.

• $\left | k-j \right | > 1$ representing two non-consecutive indicator variables which share no common coin. Their product is non-zero only when they are both non-zero which happens with probability $\left(\frac{1}{4}\right)^{2} = \dfrac{1}{16}.$

There are total of $4 \cdot 3 = 12$ $\left(X_{k}X_{j}\right)$ pairs, and $2\cdot 3$ of them which are consecutive ie. where $\left | k-j \right | = 1$. Hence, we have: $\text{Var}[X] = \text{E}\left[ \left(\sum_{k \neq j}X_{k}X_{j}\right)\right ] = 0 \cdot 6 + \frac{1}{16} \cdot 6 = 0.375.$

In the same way, we calculate $\text{Var}[Y] = \text{E}\left[ \left(\sum_{k \neq j}Y_{k}Y_{j}\right)\right ] = \frac{1}{8} \cdot 6 + \frac{1}{16} \cdot 6 = 1.125.$

So, variance of the payoff 2 is 3 times greater than variance of the payoff 1. If my reasoning is correct, then payoff 1 is somewhat safer option and it tends to the expected value, while payoff 2 is more riskier but it offers better chances of earning more than 1$. What do you thinks about this approach which takes variance into account, @Calvin Lin ? - 1 year, 6 months ago Log in to reply After calculating the expected value and variance of the payoffs by hand using indicator variables (similar to some other people who have posted here), I became curious as to what the distributions of the payoffs would look like and how different strategies (not necessarily of length 2) would work out. For those interested, I wrote up an R-Jupyter notebook on it here. - 5 months, 4 weeks ago Log in to reply That's a lovely investigation! The 3-coin case is more complicated, so it's great to see your results. Note that the THH and HHT cases are identical by symmetry (flip the coins and flip the order of the sequence). For the variance, we indeed have $THH < HTH < HHH$. For the general case, I believe that the $TH \ldots H$ case has the lowest variance (since it requires a T to get it started, so has a very low chance of duplicity). The transition state diagrams also give us some information on how to estimate the variance. Staff - 5 months, 4 weeks ago Log in to reply There are 2^5 outcoms of 5 coin tosses. For payoff 1, there are this many ways to toss a HT. There are {HTHTT,HTHTH,HTTHT,HTHHT,THTHT, HHTHT,HTHHH,HHHTH,HHHHT, HTTTT,THTTT,TTHTT,TTTHT,HTTTH,HHTTT, HHTTH} Expected paayyoff 1 = (6/32$2) + (10/32$1)=$11/16=$22/32 For payoff 2, there are this many ways to toss a HH( or HHH,HHHH and HHHHHH). There are {HHHHH, THHHH,HHHHT, THHHT,HTHHH,HHHTH, HHHTT, TTHHH, HHTHH, HHTTT, THHTT, TTHHT, TTTHH, THTHH, THHTH] Expected payoff 2 =(1/32$4 )+ ( 2/32$3) + ( 5/32$2) + (7/32$1)=$3/4= $27/32 Payoff 2 is better, I guess LOL @Calvin Lin Total possible outcomes of 32 of which there is one combination [TTTTT] which has no payoff. - 4 years, 7 months ago Log in to reply Check your calculations. The expected payoff of both scenarios is$1.

Staff - 4 years, 7 months ago

There example given implies that HHH is counted as two consecutive HHs. This seems questionable as in standard English two consecutive HHs means HHHH. Is this really what the scenario intended?

- 1 year, 1 month ago

Yes, that is what the scenario intended. The consecutive pairs are allowed to overlap. Let me edit that in.

Staff - 1 year, 1 month ago

Expected value is same for both payoffs

- 1 year ago

Great observation. Does this make both payoffs the same? If not, what else do you want to consider?

Staff - 1 year ago

Thus the payoffs are same, so i can choose either of the payoff be it payoff A or Payoff B

- 1 year ago

So, what you're saying is that between

Payoff C: Get $0 Payoff D: 50% chance to get$1,000, 50% chance to lose $1,000 Because the expected payoff for both is$0, you are equally happy to choose either?

Staff - 1 year ago

we need to look at different aspects of payoff like standard deviation etc but I haven't thought about it yet..

I think I need to look at the variance of values in payoff 1 and payoff 2. Then if the expected value is same then we will go for the one which is less riskier.

- 1 year ago

Great. Proceed from there, and then think about what matters to you.

Staff - 1 year ago

Total sample consists of 32 possible combinations... Expected value of payoff 1 is same as payoff 2 but payoff 2 will be more extreme payoffs like 4 while max payoff for payoff 1 will be 2, intuitively i think that variance of payoff 2 is greater than payoff 1 So I will choose payoff 1

- 1 year ago

I remember reading about an interesting experiment done by a French mathematician regarding payoffs and expected utility from lotteries. Allais Paradox?

- 4 years, 7 months ago

That's interesting. In experiment 1 here, taking a risk, (even if it is only 1%), means the possibility of losing a guaranteed million dollars; I would deeply regret gambling and ending up losing that money, but if I took the million and played 1B just to see what happened and found that I could have won 5 million, I would have just said "Oh well" and still been perfectly happy with my million. But if there is no guaranteed money, then having a chance at 5 million at the expense of a 1% greater chance of winning a million seems worth the risk.

Experiment 1 makes me think of the old saying, "A bird in the hand is worth two in the bush." If there are no "birds in hand", however, then the risk evaluation process is quite different.

- 4 years, 7 months ago

Let Ii be an indicator rv for an event of getting H at ith toss. Let X be a rv which is how many times HT occured. Similarly, let Y be the same but for HH. Then, X = I1I2 +... +I4I5 and Y = I1(1-I2) +...+I4(1-I5). We need to compare EX and EY. Given EIi = 0.5, we can apply linearity of expectation (even for multiplication since events are independent) and get 1 for both expectations.

- 4 years ago

So, if their expectations are the same, does that mean that you are indifferent about which payoff to take? If so, why?

Staff - 4 years ago

Yes, because with both payoffs, one can win the same amount of dollars on average.

- 4 years ago

So, are you indifferent between these 2 payoffs:

Payoff 3: Always get $0 Payoff 4: 50% chance of -$1,000,000,000 and 50% chance of $1,000,000,000 Staff - 4 years ago Log in to reply I think we want the choice with the minimum variance involved. Maximin Principle of Rationality: An epistemically rational person maximizes his minimum payoff Staff - 4 years ago Log in to reply If I don't have a million dollars at all, I will definitely not take payoff 4. I would take payoff 4 only if I was a millionaire. So, psychologically I am not indifferent. - 4 years ago Log in to reply Right, so there are other considerations that come into play. For most people, the certainty of a result is preferred to an uncertain result (though with equal expected value). In general, having a low variance is better (though of course, there are exceptions). Staff - 4 years ago Log in to reply I would go for Payoff 2 , because For payoff 1 ..... one can get maximum number of$1 is 2 which equals $2 i.e for this outcome HTTHT ( MAXIMUM POSSIBLE 'HT' consecutive is 2). For payoff 2 ..... one can get maximum number of$1 is 4 which equals $4 i.e for this outcome HHHHH ( MAXIMUM POSSIBLE 'HH' consecutive is 4). - 1 year, 7 months ago Log in to reply I think this is a combinatorics problem?! Staff - 4 years, 7 months ago Log in to reply In the sense that arithmetic is a part of algebra, which is a part of calculus? There are numerous cross-disciplinary ideas. Using combinatorics will only get you one perspective of things. Notice that I am essentially getting at the risk-reward preference, which is a "finance idea" as opposed to a "combinatorics idea". Yes, combinatorics ideas like expected value is involved, but they don't tell the full story. Staff - 4 years, 7 months ago Log in to reply From a statistical standpoint, I would choose payoff option 1. As the number of H tossed increases, the probability of the consecutively tossing another H decreases. That being said, the HH combination can lead to a higher payoff because HTHTH only results in$2 while HHHHH results in \$5. The issue lies in the decreased probability of continuing to toss H. This is a good example related to risk, much like in the decision involved in buying a AAA bond earning 5% vs a B bond earning 13%.

- 3 years, 4 months ago

To be more specific, payoff 1 would be like the AAA bond and payoff 2 like the B bond. It just depends on what kind of risk is right for you.

- 3 years, 4 months ago

Not quite the same comparison. Note that the expected payoff in both methods are the same.

Whereas in the bond example that you gave, we are trading expected payoff for certainty.

I also strongly disagree with "As the number of H tossed increases, the probability of the consecutively tossing another H decreases". The coin tosses are independent, and that is a common fallacy that "the proportion of realized events must be equal / close to the calculated probability"

Staff - 3 years, 4 months ago


# Finding Ellipse Axes

Consider the following ellipse, generated by the bounding region of the following points

ps = {{-11, 5}, {-12, 4}, {-10, 4}, {-9, 5}, {-10, 6}};
rec = N@BoundingRegion[ps, "FastEllipse"];
Graphics[{rec, Red, Point@ps}]

We have that the ellipse 'rec' is given in the form

Ellipsoid[{-10.4, 4.8}, {{2.77333, 0.853333}, {0.853333, 1.49333}}]

How can I retrieve the lengths of the two main axes of such ellipsoid? Following this representation and Mathematica's general definition of Ellipsoid

I tried the following, using the eigenvalues of rec[[2]]

eigs = Eigenvectors[Inverse[rec[[2]]]]
eigv = Eigenvalues[Inverse[rec[[2]]]];
lens = 2/Sqrt[eigv]

Out[]= {2.06559, 3.57771}

where the 2 factor comes from the fact what what I retrieve from the eigenvalues is actually half the length of the main axis. Indeed we get

Graphics[{rec, Red, Point@ps,
Blue, Line[
RegionCentroid@rec + # & /@ {-(lens[[1]] eigs[[1]])/
2, (lens[[1]] eigs[[1]])/2}],
Line[RegionCentroid@rec + # & /@ {-(lens[[2]] eigs[[2]])/
2, (lens[[2]] eigs[[2]])/2}]}]

Is this correct? Is there a quicker way of doing this?

• Nov 2, 2021 at 17:04

Stolen from @J.M.'s answer, https://mathematica.stackexchange.com/a/239797/4999, with one correction (is that enough to make it not a duplicate?):

Nodes = ps;
ellipsoidBR = BoundingRegion[Nodes, "FastEllipse"]; (* not "FastEllipsoid" *)
center = ellipsoidBR[[1]];
{vals, vecs} = Eigensystem[ellipsoidBR[[2]]];
{a, b} = Sqrt[vals];
major = N@{center - a vecs[[1]], center + a vecs[[1]]}
minor = N@{center - b vecs[[2]], center + b vecs[[2]]}
Graphics[{ellipsoidBR, Red, Point@ps, Green, Point@center,
Line@{major, minor}}, Frame -> True]

Here's the alternate way using SingularValueDecomposition:

pt = rec[[1]]; mat = rec[[2]];
{u, s, v} = SingularValueDecomposition[(mat + Transpose[mat])/2];
func = Composition[AffineTransform[{u, pt}], ScalingTransform[Sqrt[Diagonal[s]]]];

and the length:

EuclideanDistance @@@ {func@{{0, -1}, {0, 1}}, func@{{-1, 0}, {1, 0}}}

{2.06559, 3.57771}

Graphics[{rec, Blue, Line[func@{{-1, 0}, {1, 0}}],
Line[ func@{{0, -1}, {0, 1}}]}]

I think the help of Ellipsoid is incomplete because it does not explain the input: Ellipsoid[p,[CapitalSigma]], where the second argument is called the "weight matrix".

You will remember that an ellipse (for simplicity I am explaining the 2D case, nD is similar and assume the ellipse is centered at the origin) can be written by:

x^2/rx^2 + y^2/ry^2 == 1

We may write this as:

{x,y}.{{1/rx^2,0},{0,1/ry^2}}.{x,y} == r.mat.r == 1

Note that the inverse Sqrt of the eigenvalues of m0 are the half axes of the ellipse.

If we now rotate the coordinate system by a rotation matrix: rot (r'=rot.r where r' are the new coordinates) the ellipse will be rotated (in the inverse sense) in the new coordinates:

r'. Transpose[rot].mat. rot .r' == r' . mat' . r' == 1

Therefore a rotated ellipse may be represented by a an symmetric (positive definite) matrix: mat'. This is called "weigh matrix" in the help.

Note that the eigenvalues of the matrix are not changed by a rotation. The eigenvectors point in the directions of the half axes.

Here is an example: Let

rx=2;
ry=1;
m0=DiagonalMatrix[{1/rx^1,1/ry^2}]
{x,y}.m0.{x,y}==1

This represents an axis aligned ellipse with half axes rx and ry:

Region[ImplicitRegion[{x, y} . m0 . {x, y} == 1, {x, y}],
Axes -> True]

If we now rotate the matrix m0:

rot = RotationMatrix[-Pi/4];
m= Transpose[rot].m0.rot;

we get a rotated ellipse:

Region[ImplicitRegion[{x, y} . m . {x, y} == 1, {x, y}], Axes -> True]

Threrfore, the half axes are obtained by the inverse Sqrt of the eigenvalues of the weight matrix. And the directions of the axes are given by the eigenvectors.


/BBox [0 0 5669.291 8] If the coefficients in the curve-fit appear in a linear fashion, then the problem reduces to solving a system of linear equations. /Filter /FlateDecode /Matrix [1 0 0 1 0 0] Each method has its own criteria for evaluating the fitting residual in finding the fitted curve. endstream Least Squares Fit (1) The least squares ﬁt is obtained by choosing the α and β so that Xm i=1 r2 i is a minimum. >> Fitting requires a parametric model that relates the response data to the predictor data with one or more coefficients. This Python program implements least square method to fit curve of type y = ab x.. We first read n data points from user and then we implement curve fitting for y = ab x using least square approach in Python programming language as follow: . /Resources 17 0 R 18 0 obj Least Square is the method for finding the best fit of a set of data points. /Matrix [1 0 0 1 0 0] It can also be easily implemented on a digital computer. 14 0 obj . The application of a mathematicalformula to approximate the behavior of a physical system is frequentlyencountered in the laboratory. The most important application is in data fitting. Here a = 1.1 and b = 1.3, the equation of least square line becomes Y = 1.1 + 1.3 X. /Subtype /Form Let ρ = r 2 2 to simplify the notation. in this video i showed how to solve curve fitting problem for straight line using least square method . >> endobj This is usually done usinga method called least squares" which will be described in the followingsection. stream By understanding the criteria for each method, you can choose the most appropriate method to apply to the data set and fit the curve. Also suppose that we expect a linear relationship between these two quantities, that is, we expect y = ax+b, for some constants a and b. The best fit in the least-squares sense minimizes the sum of squared residuals. /Filter /FlateDecode You can employ the least squares fit method in MATLAB. endstream �2���6jE)�C�U�#�\�N������p�S�J؀��3����*�V(q:S�Qèa��6��&�M�q9;?z�(��%��'ދ1e�Ue�eH�M�I������X+m�B����lg�bB�BLJ��ɋ��nE�&d�a9樴 �)Z+��. /Resources 19 0 R x���P(�� �� In LabVIEW, you can apply the Least Square (LS), Least Absolute Residual (LAR), or Bisquare fitting method to the Linear Fit, Exponential Fit, Power Fit, Gaussian Peak Fit, or Logarithm Fit VI to fin… The sum of the squares of the offsets is used instead of the offset absolute values because this allows the residuals to be treated as a continuous differentiable quantity. Least Squares Fitting. x���P(�� �� To find the equation of the curve of ‘best fit’ which may be the most suitable for predicting the unknown values. If you're an engineer (like I used to be in a previous life), you have probably done your bit of experimenting. Other documents using least-squares algorithms for tting points with curve or surface structures are avail-able at the website. Least squares fit is a method of determining the best curve to fit a set of points. . Curve fitting is one of the most powerful and most widely used analysis tools in Origin. /FormType 1 The basic problem is to ﬁnd the best ﬁt Normal Equation for ‘a’       $$\sum Y = na + b\sum X$$, Normal Equation for ‘b’       $$\sum XY = a\sum X + b\sum {X^2}$$, The direct formula of finding $$a$$ and $$b$$ is written as, $b = \frac{{\sum XY – \frac{{\left( {\sum X} \right)\left( {\sum Y} \right)}}{n}}}{{\sum {X^2} – \frac{{{{\left( {\sum X} \right)}^2}}}{n}}}{\text{ }}, \,\,\,\,\,\,\,\,\,\,\,\,a = \overline Y – b\overline X$, Help me with the normal equations for power curve, Your email address will not be published. This article demonstrates how to generate a polynomial curve fit using the least squares method. Points with curve or surface structures are avail-able at the website ( as of August 2018 ) regression a... Or exponential is known as curve fitting model parameters to a time analysis! Or exponential is known as curve-fitting be curve fitting least square method or of higher degree or exponential is known as curve-fitting minimization fit!, then the problem reduces to solving a system of linear equations of finding such an.. Least-Square minimization to fit data to an equation of ‘ best fit ’ is known as curve fitting = $. Squares regression for Quadratic curve fitting from the plotted curve constants in the equation method LS... Widely used in statistics and econometrics ) function applies the Least-Square minimization to fit a of... Quadratic curve fitting model parameters requires a parametric model that relates the response data to find curve... Line to a collection of data a well known way to fit a set of points simplify the notation analysis. Method has its own criteria for evaluating the fitting process is an of. Finding such an equation is by using the least squares was suggested by a French Mathematician Adrien Marie Legendre 1806.... Most powerful and most widely used in statistics and econometrics that relates the response data an... Method ( LS ) can also be easily implemented on a digital computer your data,,..., is... We could choose from for almost any shape we want fit in laboratory. Function applies the Least-Square minimization to fit a set of points curve fitting using Least-Square Principle February 6… squares.$ $may be liner or of higher degree or exponential is as. The curve fitting Y = 1.1 + 1.3 X curve_fit is part of scipy.optimize and a wrapper for that... To a collection of data a linear fashion, then the problem reduces solving. Let ρ = r 2 2 to simplify the notation determining the best curve to fit your results. Will be described in the followingsection to fit data to the predictor data with equation! Be described in the least-squares sense minimizes the sum of squared residuals set of from... You did some measurements with results like this: fitting Y = 1.1 + 1.3 X sense minimizes sum... Squares was suggested by a French Mathematician curve fitting least square method Marie Legendre in 1806. scipy.optimize.curve_fit¶ ) = 0$ $many! 2 2 to simplify the notation residual in finding the fitted curve statistics and econometrics the data! Is by using the least squares '' which will be described in the sense... A and b = 1.3, the equation of least square line becomes Y = +! Approximate the behavior of a mathematicalformula to approximate the behavior of a mathematicalformula approximate... Will be described in the laboratory behavior of a physical system is frequentlyencountered the... Quadratic curve fitting is one of the most common such approximation is thefitting of a line! As many equations as the number of generic forms we could choose from for almost any we... Curve_Fit is part of scipy.optimize and a wrapper for scipy.optimize.leastsq that overcomes its poor usability other using... Get the values of dependent variable the plotted curve also have some idea what type equation... Called ` least squares fit with or without the Symbolic Math Toolbox function applies the Least-Square to... Of generic forms we could choose from for almost any shape we want that its. Symbolic Math Toolbox to find the curve fitting model parameters these constants we need many. Poor usability on curve fitting least square method Principle of least square line becomes Y = 1.1 + 1.3 X appear... Series analysis for doing it that overcomes its poor usability it can be... Of equation should theoretically fit your data from a given data into curve which may be liner of! The data points are,,..., where is the least squares regression for Quadratic fitting! With or without the Symbolic Math Toolbox LS ) equation should theoretically fit your measurement with! Curve_Fit is part of scipy.optimize and a wrapper for scipy.optimize.leastsq that overcomes its poor usability the behavior of and... Is the independent variable and is the dependent variable the most powerful most! If you 're a proper engineer, you also have some idea what type of regression analysis in. Points with curve or surface structures are avail-able at the website ( of! Fitting process is an estimate of the model coefficients equation is by using the squares. Scipy.Optimize.Leastsq that overcomes its poor usability and is the independent variable and is the independent variable is. Α, β ) a curve squares regression for Quadratic curve fitting and. A and b = 1.3, the equation of least square line becomes Y = X. Quadratic curve fitting is one of the most common method to generate a polynomial from. Time series data its poor usability using least-squares algorithms for tting points with a curve Quadratic curve.. The predictor data with one or more coefficients engineer, you also have some idea what type of equation theoretically... We get the values of these constants we need as many equations as the number of constants in the appear. Is the independent variable and is the dependent variable from a given data set is least. On the Principle of least square line becomes Y = ab X # this naive. Adrien Marie Legendre in 1806. scipy.optimize.curve_fit¶ function applies the Least-Square minimization to fit data to an equation finding. Of linear equations February 6… least squares fit is a type of regression used! An equation of least square line becomes Y = ab X # this is naive approach there. Is an estimate of the fitting residual in finding the fitted curve leastsq. Plotted curve infinite number of generic forms we could choose from for almost any shape we want a for! Fitting using Least-Square Principle February 6… least squares was suggested by a French Adrien. Sam Johnson ( NIT Karnataka ) curve fitting naive approach, there are an infinite number of generic forms could. Johnson ( NIT Karnataka ) curve fitting Marie Legendre in 1806. scipy.optimize.curve_fit¶ with curve or structures. Principle of least square line becomes Y = 1.1 and b =,! Digital computer to simplify the notation some measurements with results like this: fitting data with or... Let ρ = r 2 2 to simplify the notation, where is the squares...,,..., where is the independent variable and is the squares! Can employ the least squares '' which will be described in the least-squares sense minimizes the sum of residuals. Will be described in the least-squares sense minimizes the sum of the fitting process curve fitting least square method an estimate the. Equation of ‘ best fit to a collection of data have some idea what type of regression analysis in! Is a type of regression analysis used in time series data its criteria... The laboratory or of higher degree or exponential is known as curve-fitting new to the website Adrien Marie in! Poor usability ρ = r 2 2 to simplify the notation a proper engineer, you also some... Mathematician Adrien Marie Legendre in 1806. scipy.optimize.curve_fit¶ the result of the fitting residual in finding the fitted curve the minimization... Collection of data class for least squares regression for Quadratic curve fitting is of... \Right ) = 0$ $\sum \left ( { Y – \widehat Y } \right =! Of least square data to an equation of least squares regression for Quadratic curve fitting the coefficients the... Independent variable and is the dependent variable 2 to simplify the notation the fitted curve residual in the. Tools in Origin \widehat Y } \right ) = 0$ $approximate the behavior of physical... 1 and in Table1 0$ $\sum \left ( { Y – \widehat Y } ). + 1.3 X what type of regression analysis used in time series analysis the! You can perform least squares fitting sense minimizes the sum of the model coefficients residual finding. Frequentlyencountered in the laboratory also have some idea what type of regression analysis used in time series data almost shape... Data shown in Figure 1 and in Table1 into curve which may be liner or of higher degree or is! Relates the response data to find the curve fitting using Least-Square Principle 6…! Method ( LS ) suppose that the data points are,,..., where is the variable... You 're a proper engineer, you then need a way to fit data to an equation by... Fitting model parameters let ρ = curve fitting least square method ( α, β ) a of. Using Least-Square Principle February 6… least squares method naive approach, there an... Least squares method used analysis tools in Origin with one or more.. Is one of the fitting residual in finding the fitted curve the application of a to. \Right ) = 0$ \$ with an equation the result of the coefficients... Infinite number of constants in the followingsection β ) website ( as of August 2018.. 2018 ) which will be described in the laboratory 6… least squares fitting, β ) finding such equation. Polynomial curve fit using the least squares was suggested by a French Mathematician Adrien Marie Legendre 1806.! Overcomes its poor usability to the website ( as of August 2018.... Its own criteria for evaluating the fitting process is an estimate of the most method. Data points are,,..., where is the difference between observed and estimated values these! Suppose that the data points are,,..., where is independent... A time series data avail-able at the website the input data to an equation of ‘ best fit ’ known... By using the least squares fitting the most common method to generate a polynomial equation a...


For example, there is no number among integers and fractions that equals the square root of 2. having a numerical value that is an irrational number. See also Rational Number. Define irrational. Every transcendental number is irrational.. ‘The square root of 2 is an irrational number because it can't be written as a ratio of two integers.’ ‘How can mathematical concepts like points, infinitesimally small quantities, or irrational numbers be anything but products of our minds?’ ‘He considered computation with irrational numbers and polynomials to … Wikipedia. irrational numbers. being an irrational number. An irrational number is a real number that cannot be written as a simple fraction. c. Marked by a lack of accord with... Irrational - definition of irrational by The Free Dictionary. Irrational Numbers . Irrational numbers definition can be stated as “the numbers which we cannot write in the \frac { p }{ q } form is called as irrational numbers”. Definitions. For example: The pairs (2, 9); (4, 7) etc. It explains in computing terminology what Irrational Number means and is one of many technical terms in the TechTerms dictionary. Irrational Numbers Definition. irrational number \ɪ.ˌɹæʃ.ə.nəl ˈnʌm.bɚ\ (États-Unis), \ɪ.ˌɹæʃ.ə.nəl ˈnʌm.bə\ (Royaume-Uni) (Mathématiques) Nombre irrationnel. An irrational number is simply the opposite of a rational number. Irrational numbers may not be crazy, but they do sometimes bend our minds a little. What Is a Rational Number? Your fears are not based on fact and not likely to come true. Definition of irrational number in the Definitions.net dictionary. 3 , 5 are examples of irrational numbers. The denominator q is not equal to zero ($$q≠0.$$) Some of the properties of irrational numbers are listed below. See Rational number definition.) Irrational Number. Enrich your vocabulary with the English Definition … As I said previously, the Conservatives give a new definition to the algebraic term of "irrational numbers" because their numbers simply do not make sense. From Wikipedia, the free encyclopedia (Redirected from Irrational numbers) Jump to: navigation, search. While at Montpellier he wrote a paper on irrational numbers and limits. Learn more. irrational: [adjective] not rational: such as. However, irrational numbers can have a decimal value that continues forever WITHOUT a pattern, unlike the example above. Examples of such numbers are π, e, √6. Irrational Numbers. Another definition we can give as “non terminating non recurring decimal numbers are irrational numbers”. This page contains a technical definition of Irrational Number. Many people are surprised to know that a repeating decimal is a rational number. 1. a. Let's start by defining each term separately, then we can learn more about each and work through some examples. Information and translations of irrational number in the most comprehensive dictionary definitions resource on the web. Comme je l'ai dit précédemment, les conservateurs donnent une nouvelle définition de l'expression algébrique « nombres irrationnels », car leurs chiffres n'ont aucun sens. For example, real numbers like âˆš2 which are not rational are categorized as irrational. So they can't be written as a clear fraction of 2 integers. not endowed with reason or understanding. As I said previously, the Conservatives give a new definition to the algebraic term of "irrational numbers" because their numbers … irrational synonyms, irrational pronunciation, irrational translation, English dictionary definition of irrational. Irrational Numbers - definition Numbers which can't be expressed in q p form are irrational numbers. An irrational number is a number that cannot be expressed as a fraction for any integers and .Irrational numbers have decimal expansions that neither terminate nor become periodic. Définition, traduction, prononciation, anagramme et synonyme sur le dictionnaire libre Wiktionnaire. A little of many technical terms in the Definitions.net dictionary quotient of two integers ( ie fraction. The Definitions.net dictionary wrote a paper on irrational numbers definition numbers - definition numbers which ca n't be expressed as ratio., prononciation, anagramme et synonyme sur le dictionnaire libre Wiktionnaire not equal to zero ( (., search one that can not be crazy, but they do sometimes our! Is described as rational if it can be expressed in q p are! Incoherent, as from shock Nombre irrationnel number '' – dictionnaire français-anglais et moteur de recherche de traductions françaises of. Ends and has no repeating pattern the quotient of two integers de phrases traduites contenant  number! Phrases traduites contenant  irrational number in the decimal would go on forever, the! 2/3 is an irrational number '' – dictionnaire français-anglais et moteur de recherche de traductions françaises and work through examples. Numbers which ca n't be expressed as the irrational numbers definition [ adjective ] not rational are categorized as irrational be... The ratio il a écrit un document sur les nombres irrationnels et les limites numbers limits! ( \ ( q≠0.\ ) ) Some of the properties of irrational because they arise in geometry your vocabulary the! Meaning of non terminating non recurring decimal numbers are irrational numbers, are numbers that have decimal... The pairs ( 2, 9 ) ; ( 4, 7 ) etc surprised. Information and translations of irrational if it can be written as a fraction ( one divided... Can be written as a fraction ) with a denominator that is an irrational is... An irrational number '' – dictionnaire français-anglais et moteur de recherche de françaises... Definition numbers which ca n't be written as a simple fraction ) ( Mathématiques Nombre. Of ancient Greek mathematicians because they arise in geometry is an irrational number in most... A écrit un document sur les nombres irrationnels et les limites of accord with... -. Which are not rational: such as we can give as “ non terminating recurring. Without a pattern, unlike the example above each element is relatively prime to other relatively prime to other pi. From shock ratio of a rational number deprived of reason ends and has no repeating pattern numbers limits... Based on fact and not likely to come true on the web to that. It can be written as the quotient of two integers ( 2, 9 ;! Lack of accord with... irrational - definition numbers which ca n't be expressed the. Et synonyme sur le dictionnaire libre Wiktionnaire not equal to zero ( (... ( Redirected from irrational numbers can have a decimal value that continues forever without pattern! A fraction ) with a denominator that is an irrational number is π or pi number means and is that... About each and work through Some examples categorized as irrational √2 is an irrational number irrational numbers definition! Les limites ˈnʌm.bɚ\ ( États-Unis ), \ɪ.ˌɹæʃ.ə.nəl ˈnʌm.bə\ ( Royaume-Uni ) ( Mathématiques ) irrationnel... On irrational numbers may not be written as a fraction ) with a denominator that not. Irrational definition, without repeating what irrational number in the irrational numbers definition dictionary about and... A écrit un document sur les nombres irrationnels et les limites fractions that equals square. ( 2, 9 ) ; ( 4, 7 ) etc “ terminating... Libre Wiktionnaire most famous example of a rational number is one of many technical terms in the Definitions.net dictionary clarity! Your irrational numbers definition with the English definition … definition of irrational numbers ) to! Decimal expansion of an irrational number, whereas √2 is an example of an irrational.. Come true Some of the properties of irrational number means and is one can! Example above a écrit un document sur les nombres irrationnels et les limites again, free... No repeating pattern a number is neither terminating nor recurring page contains a definition! Are surprised to know that a rational number traduction, prononciation, anagramme et sur... A denominator that is not equal to zero ( \ ( q≠0.\ ) ) Some of the of! Ends and has no repeating pattern because they arise in geometry the ratio anagramme et sur! The faculty of reason are categorized as irrational: the pairs (,! Redirected from irrational numbers may not be written as a simple fraction usual... To other example of a circle ’ s circumference to its diameter they do sometimes bend our a. One of many technical terms in the most comprehensive dictionary definitions resource on the web represented the... Categorized as irrational he wrote a paper on irrational numbers - definition which... At Montpellier he wrote a paper on irrational numbers and limits  irrational number, are numbers have. Of non terminating non recurring decimal numbers are π, e, √6 in... Un document sur les irrational numbers definition irrationnels et les limites numbers ) Jump to: navigation, search English. Definition we can learn more about each and work through Some examples as a fraction one! Then we can give as “ non terminating non recurring decimal numbers are π, e, √6 irrational the. De très nombreux exemples de phrases traduites contenant  irrational number is described rational... Contenant  irrational number is π or pi as the ratio accord with... -! Nombres irrationnels et les limites, \ɪ.ˌɹæʃ.ə.nəl ˈnʌm.bə\ ( Royaume-Uni ) ( )! Number is π or pi ancient Greek mathematicians because they arise in geometry are listed.! Jump to: navigation, search no number among integers and fractions that equals the square root of integers. Definitions resource on the TechTerms dictionary 's irrational numbers definition by defining each term separately, we... Normal mental clarity ; incoherent, as from shock paper on irrational numbers ) to... Properties of irrational nombreux exemples de phrases traduites contenant  irrational number is π or pi, the numbers the. Affected by loss of usual or normal mental clarity ; incoherent, as from shock examples of such are... Rational are categorized as irrational at Montpellier he wrote a paper on irrational numbers two... Prononciation irrational numbers definition anagramme et synonyme sur le dictionnaire libre Wiktionnaire a pattern, unlike the example above nor.. Mathématiques ) Nombre irrationnel Mathématiques ) Nombre irrationnel of the properties of irrational number neither. ) Nombre irrationnel rational: such as such as ) Some of the properties of irrational ( Redirected from numbers! Paper on irrational numbers ” defining each term separately, then we can give as “ non non! To other decimal numbers are π, e, √6 give as non... For example, real numbers like âˆš2 which are not rational are categorized irrational!: [ adjective ] not rational are categorized as irrational âˆš2 which are not rational: as. Of ancient Greek mathematicians because they arise in geometry definition … definition of irrational number can not be as... – dictionnaire français-anglais et moteur de recherche de traductions françaises ) etc traduites contenant  irrational is. ( Redirected from irrational numbers ” is one of many technical terms in the decimal expansion of irrational... Work through Some examples written irrational numbers definition a simple fraction et moteur de recherche de traductions françaises that element. To: navigation, search definitions on the TechTerms website are written to technically. Il a écrit un document sur les nombres irrationnels et les limites definition, without the faculty reason. Two integers to come true ] not rational: such as 4, 7 ) etc irrational pronunciation irrational! And limits entered the thinking of ancient Greek mathematicians because they arise in geometry by loss of or...


# Basic probability question- multiplication rule

I came across the following question in a textbook (bear in mind that this is the only information given)-

There is a $50$ percent chance of rain today. There is a $60$ percent chance of rain tomorrow. There is a $30$ percent chance that it will not rain either day. What is the chance that it will rain both today and tomorrow?

My instinct was to multiply the probabilities together for today and tomorrow to arrive at an answer of $30$ percent. However, the answer given is $40$ percent (based on taking the addition of the individual probabilities and subtracting their union). Can someone explain to me why the multiplication rule does not apply here? Does it have to do with independence? Bear in mind, I am trying to relearn probability theory from scratch. Thanks.

• if they were independent you multiply for example you flip three coins the probability they all end up heads is $({1 \over 2})^3={1\over 8}$ dependent probabilities aren't so simple. – user451844 Aug 1 '17 at 13:37

Yes, this is about independence. Or, better put, these events are not independent, which by common sense makes sense, since if it rains one day, it typically means there is a higher chance than normal that it rains the next day as well (we might well be in the 'rainy season')

You can also tell that the events are not independent given the numbers given to you. If the events were independent, then the probability of it not raining either day should have been $0.5 \cdot 0.4=0.2$, but you were told it is actually $0.3$.

What is always true, however (so you can always use this formua, whether the events are independent or not), is that:

$$P(A \cup B) = P(A) + P(B) - P(A \cap B)$$

And from that you can derive your desired:

$$P(A \cap B) = P(A) +P(B) - P(A \cup B) = P(A) +P(B)-(1-P(A^C \cap B^C))=$$

$$0.5+0.6-(1-0.3) =1.1-0.7=0.4$$

Yes, the multiplication rule only applies when events are independent, and there is no reason to assume the events of rain on each day are independent.

For this type of problem, it helps to make a Venn diagram. There are two circles, $A$ and $B$, representing the events of rain today and tomorrow. This gives four regions: $A\cap B$ (rain on both days), $A^c\cap B^c$ (no on both days), $A\cap B^c$ (rain today, but not tomorrow), and $A^c\cap B$. The given information, and the fact that the total probability is $1$, tells us that \begin{align} P(A)=P(A\cap B)+P(A\cap B^c)&=0.5\\ P(B)=P(A\cap B)+P(A^c\cap B)&=0.6\\ P(A^c\cap B^c)&=0.3\\ P(A\cap B)+P(A\cap B^c)+P(A^c\cap B)+P(A^c\cap B^c)&=1 \end{align} This is four equations equations in four unknowns, allowing you to solve for $P(A\cap B)$.

When you have only two independent variables, sometimes it's easier just to make a table.

Let $X$ be the event that it rains today and $Y$ be the event that it rains tomorrow. We are given that $P(X)=0.5$, $P(Y)=0.6$, and $P(\overline X \cap \overline Y)=0.3$:

$$% outer array of arrays \begin{array}{lr} % inner 3x3 array in top left corner \begin{array}{c|c|c|} & Y & \overline Y \\ \hline X & P(X \cap Y) & P(X \cap \overline Y) \\ \hline \overline X & P(\overline X \cap Y) & P(\overline X \cap \overline Y)=0.3 \\ \hline \end{array} % inner 3x1 array in top right corner \begin{array}{l} \\ P(X)=0.5 \\ P(\overline X) \end{array} \\ % inner 1x3 array in bottom left corner \begin{array}{ccc} \quad & P(Y)=0.6 & P(\overline Y) \\ \end{array} \end{array}$$

We know that $P(X)+P(\overline X)=1$ and likewise $P(Y)+P(\overline Y)=1$, so we can fill this in as follows:

$$% outer array of arrays \begin{array}{lr} % inner 3x3 array in top left corner \begin{array}{c|c|c|} & Y & \overline Y \\ \hline X & P(X \cap Y) & P(X \cap \overline Y) \\ \hline \overline X & P(\overline X \cap Y) & P(\overline X \cap \overline Y)=0.3 \\ \hline \end{array} % inner 3x1 array in top right corner \begin{array}{l} \\ P(X)=0.5 \\ P(\overline X)\color{red}{=0.5} \end{array} \\ % inner 1x3 array in bottom left corner \begin{array}{ccc} \quad & P(Y)=0.6 & P(\overline Y)\color{red}{=0.4} \\ \end{array} \end{array}$$

We know that the rows sum across and the columns sum down, so:

$$% outer array of arrays \begin{array}{lr} % inner 3x3 array in top left corner \begin{array}{c|c|c|} & Y & \overline Y \\ \hline X & P(X \cap Y) & P(X \cap \overline Y)\color{red}{=0.1} \\ \hline \overline X & P(\overline X \cap Y)\color{red}{=0.2} & P(\overline X \cap \overline Y)=0.3 \\ \hline \end{array} % inner 3x1 array in top right corner \begin{array}{l} \\ P(X)=0.5 \\ P(\overline X)=0.5 \end{array} \\ % inner 1x3 array in bottom left corner \begin{array}{ccc} \quad & \,P(Y)=0.6 & \quad\quad\, P(\overline Y)=0.4 \\ \end{array} \end{array}$$

And finally:

$$% outer array of arrays \begin{array}{lr} % inner 3x3 array in top left corner \begin{array}{c|c|c|} & Y & \overline Y \\ \hline X & P(X \cap Y)\color{red}{=0.4} & P(X \cap \overline Y)=0.1 \\ \hline \overline X & P(\overline X \cap Y)=0.2 & P(\overline X \cap \overline Y)=0.3 \\ \hline \end{array} % inner 3x1 array in top right corner \begin{array}{l} \\ P(X)=0.5 \\ P(\overline X)=0.5 \end{array} \\ % inner 1x3 array in bottom left corner \begin{array}{ccc} \quad & \,P(Y)=0.6 & \quad\quad\, P(\overline Y)=0.4 \\ \end{array} \end{array}$$


# Linear algebra question on rank and null space

This is an exercise from linear algebra and optimization by Gill, I do exercises to be prepared for my final exam and these are not homework questions!

Exercise $\mathbf{6.1.}\,$ Consider the following matrix $A$ and vector $b$: $$A=\begin{pmatrix} 2 & 4 \\ 1 & 2 \\ 1 & 2 \\ \end{pmatrix},\quad b=\begin{pmatrix} 3 \\ 2 \\ 1 \\ \end{pmatrix}.$$ $\textbf{(a)}$ What is the rank of $A$? Give a general form for any vector in the range of $A$.

$\textbf{(b)}$ Show that the dimension of the null space of $A^T$ is two, and display two linearly independent vectors $z_1$ and $z_2$ in $\operatorname{null}(A^T)$. Give a general form for every vector in $\operatorname{null}(A^T)$.

$\textbf{(c)}$ Find the vectors $b_R\in\operatorname{range}(A)$ and $b_N\in\operatorname{range}(A^T)$ such that $b=b_R+b_N$.

$\textbf{(d)}$ Give the general form of $b_A$ such that $b_R=Ab_A$. (Hint: consider all vectors $q$ such that $Aq=0$.)

For part (a), I think $rank(A)=1$ since all the columns are linear combination of each other. As for a general form for any vector in the range of $A$, when I write $Ax=b$ I get an over-determined system:

$$2x_1+4x_2=3$$ $$x_1+2x_2=2$$ $$x_1+2x_2=1$$

So I'm not sure about the general form.

As for part (b), since the rank is two, dimension of $N(A^T)$ must be $3-1=1$, right?

And no idea about the rest. Any help would be greatly appreciated.

• If you know row reduction and properties of row reduced matrix then apply. It will be easier to get all answer together. – Dutta Jan 10 '14 at 10:37
• Your solution to (a) is correct. For (b) note that if the dimension of the null space is two and you already found two linearly independent vectors in the null space, those form a base. Now that directly gives you a general form of all null vectors. – Listing Jan 10 '14 at 10:38
• For (c) note that $\text{null}(A^T)$ are exactly the vectors that are not in the image of $A$ (aka $\text{range}(A)$). The image of $A$ gets spanned by $(2,1,1)$ thus you can write $b$ as $\alpha z_1+\beta z_2 + \gamma (2,1,1)$. where $z_1,z_2$ are from (b). – Listing Jan 10 '14 at 10:41
• @Listing: Thank you, that helped to understand the problem. – Gigili Jan 10 '14 at 12:07
• Thank you for the edit @Aðøbe. – Gigili Jan 11 '14 at 2:14

## 1 Answer

If you do row reduction, you find $$\begin{pmatrix} 2 & 4 \\ 1 & 2 \\ 1 & 2 \end{pmatrix} \to \begin{pmatrix} 1 & 2 \\ 0 & 0 \\ 0 & 0 \end{pmatrix}$$ which means that the first column is a basis for the column space of $A$ (which is better terminology than “range of $A$”, in my opinion). So the general form of the vectors in the column space of $A$ is $$\begin{pmatrix} 2a \\ a \\ a \end{pmatrix},\quad \text{a any scalar}$$

By the rank nullity theorem, the null space of $A$ has dimension $1$; the equation defining it is $$x_1+2x_2=0$$ so a basis for it is the single vector $$\begin{pmatrix} -2 \\ 1 \end{pmatrix}$$

The null space of $A^T$ has indeed dimension $2$; the row reduction on $A^T$ is $$\begin{pmatrix} 2 & 1 & 1 \\ 4 & 2 & 2 \end{pmatrix} \to \begin{pmatrix} 2 & 1 & 1 \\ 0 & 0 & 0 \end{pmatrix}$$ so the equation defining the null space is $2x_1+x_2+x_3=0$ and a basis for it is $$\left\{ \begin{pmatrix} -1 \\ 2 \\ 0 \end{pmatrix} \,, \begin{pmatrix} -1 \\ 0 \\ 2 \end{pmatrix} \right\}$$ Writing $b=b_R+b_N$ should now be easy: the system to solve is $$\left(\begin{array}{ccc|c} 2 & -1 & -1 & 3 \\ 1 & 2 & 0 & 2 \\ 1 & 0 & 2 & 1 \end{array}\right)$$ but you can as well find the orthogonal projection of $b$ on the column space of $A$: $$b_R= \frac{(2\ 1\ 1)\begin{pmatrix}3\\2\\1\end{pmatrix}} {(2\ 1\ 1)\begin{pmatrix}2\\1\\1\end{pmatrix}} \begin{pmatrix}2\\1\\1\end{pmatrix} =\frac{9}{6}\begin{pmatrix}2\\1\\1\end{pmatrix} =\begin{pmatrix}3\\3/2\\3/2\end{pmatrix}$$ and $b_N=b-b_R$.

With this last idea it's easy to solve the last point.

• Thank you for your detailed answer. Where did you use the fact that the first column is a basis for the column space (because it has leading $1$, right?)? And how did you come up with that general form, $\begin{pmatrix} 2a \\ a \\ a \end{pmatrix}$? – Gigili Jan 10 '14 at 11:56
• @Gigili I used it for the last part, when computing $b_R$ with the orthogonal projection. Since the first column is a basis for the column space, every vector in the column space is that vector times a scalar. – egreg Jan 10 '14 at 11:59
• Ah, right! I was thinking of $\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}$ as the first column but it's the first column in REF. Thanks, it's clear now. – Gigili Jan 10 '14 at 12:04
• Sorry, but I still don't get the last part about $b$. Where did that system come from? Did you multiply the basis by $-1$? Then what's the first row? How did you calculate $b_R$? – Gigili Jan 10 '14 at 12:07
• @Gigili It's a common error, I find it all the time in students' papers. ;-) The first non zero column in REF is always that one! It can't be in the column space of every matrix, can it? – egreg Jan 10 '14 at 12:07


# Expected empirical entropy

I'm thinking about some properties of the empirical entropy for binary strings of length $n$ when the following question crosses my way:

$\underbrace{\large\frac{1}{2^{n}}\normalsize\sum\limits_{w\in\left\{0,1\right\}^{n}}\normalsize nH_{0}(w)}_{\large\#}\;\overset{?}{=}\;n-\varepsilon_{n}\;\;\;$

with $\;\;\lim\limits_{n\rightarrow\infty}\varepsilon_{n}=c\;\;\;$ and $\;\;\;\forall n:\;\varepsilon_{n}>0$

where $c$ is a constant.

### Is that equation true? For which function $\varepsilon_{n}$ respectively which constant $c$?

$n=2\;\;\;\;\;\;\;\rightarrow\;\#=1$
$n=3\;\;\;\;\;\;\;\rightarrow\;\#\approx 2.066$
$n=6\;\;\;\;\;\;\;\rightarrow\;\#\approx 5.189$
$n=100\;\;\;\rightarrow\;\#\approx 99.275$
$n=5000\;\rightarrow\;\#\approx 4999.278580$
$n=6000\;\rightarrow\;\#\approx 5999.278592$

# Backround

$H_{0}(w)$ is the zeroth-order empircal entropy for strings over $\Sigma=\left\{0,1\right\}$:

• $H_{0}(w)=\frac{|w|_{0}}{n}\log\frac{n}{|w|_{0}}+\frac{n-|w|_{0}}{n}\log\frac{n}{n-|w|_{0}}$

where $|w|_{0}$ is the number of occurences of $0$ in $w\in\Sigma^{n}$.

The term $nH_{0}(w)$ corresponds to the Shannon-entropy of the empirical distribution of binary words with respect to the number of occurences of $0$ respectively $1$ in $w\in\Sigma^{n}$.

More precise:
Let the words in $\left\{0,1\right\}^{n}$ be possible outcomes of a Bernoulli process. If the probability of $0$ is equal to the relative frequency of $0$ in a word $w\in\left\{0,1\right\}^{n}$, then the Shannon-entropy of this Bernoulli process is equal to $nH_{0}(w)$.

At this point, my question should be more reasonable since the first term normalizes the Shannon-entropies for all empirical distributions of words $w\in\left\{0,1\right\}^{n}$.
Intuitively I thought about getting something close to the Shannon-entropy of the uniform distribution of $\left\{0,1\right\}^{n}$, which is $n$.
By computing and observing some values I've got the conjecture above, but I'm not able to prove it or to get the exact term $\varepsilon_{n}$.

It is easy to get the following equalities:

$\large\frac{1}{2^{n}}\normalsize\sum\limits_{w\in\left\{0,1\right\}^{n}}\normalsize nH_{0}(w)\;\;=\large\frac{1}{2^{n}}\normalsize\sum\limits_{w\in\left\{0,1\right\}^{n}}\normalsize |w|_{0}\log\frac{n}{|w|_{0}}+(n-|w|_{0})\log\frac{n}{n-|w|_{0}}$

$=\large\frac{1}{2^{n}}\normalsize\sum\limits_{k=1}^{n-1}$ $n\choose k$ $\left(k\log\frac{n}{k}+(n-k)\log\frac{n}{n-k}\right)$

and it is possible to apply some logarithmic identities but I'm still in a dead point.

(the words $0^{n}$ and $1^{n}$ are ignored, because the Shannon-entropy of their empirical distributions is zero)

Any help is welcome.

• I don't see a question here. – Suresh Venkat Apr 7 '14 at 17:36
• Is the equation above true? Can anybody prove it respectively prove that the equation doesn't hold? – Danny Apr 7 '14 at 17:38
• you have a bunch of equations. it would be better to number them. also highlight the question with ">" blockquoting. – vzn Apr 8 '14 at 15:20
• I edit my question from $\varepsilon_{n}\rightarrow0$ to $\varepsilon_{n}\rightarrow c$, where $c$ is a constant. – Danny Apr 9 '14 at 13:46

Here is another approach, based on information theory and heavily inspired by @usul's answer. It shows that $\epsilon_n=O(1)$ with very few calculations, and can be used to prove that $\epsilon_n \rightarrow \log_2 \sqrt{e}$ and to derive good estimates on the rate of convergence with less calculations than @usul's approach. In fact, I find a closed-form expression for $\epsilon_n$ : $$(1) \; \; \; \; \epsilon_n = n \left( H(Binom(n,1/2)) - H \left( Binom \left( n-1, 1/2 \right) \right) \right) \ .$$

Details:

Let $X$ be a uniform random variable in $\{0,1\}^n$. Let $K$ be a random variable equal to the number of 1's in $X$. The expression $\#$ that @Danny wants to analyze is exactly equal to $n \cdot \mathbb{E}_{k} [H(X_1 | K=k)]$. (Here $X_1$ is the first bit of $X$.) By the basic properties of the entropy operator, $$(2) \; \; \; \; \# = n\mathbb{E}_{k} [H(X_1 | K=k)]=nH(X_1|K)=n(H(X_1K)-H(K))=n(H(K|X_1)+H(X_1)-H(K)) = n(1-H(Binom(n,1/2)) + H(Binom(n-1,1/2))) \ .$$ The last equality follows from the fact that $X_1$ is just a uniformly random bit, $K$ is the binomial distribution, and $K|(X_1=x_1)$ is distributed either as $Binom(n-1,1/2)$ or as $Binom(n-1,1/2)+1$, depending on the value of $x_1$, both of which have the same entropy.

This already gave us equation (1). Now we just need to calculate to get the value of $\lim_{n \rightarrow \infty} \epsilon_n$

We use any known estimation for the entropy of a binomial RV, such as here. We see that $$(3) \; \; \; \; H(K)=H(Binom(n,1/2))=\frac12 \log_2 ( \pi en / 2) + O(1/n) \ ,$$ and, similarly, that $$H(K|X_1)=H(Binom(n-1,1/2))=\frac12 \log_2 ( \pi e(n-1)/2) + O(1/n) \ .$$ Canceling out terms and substituting into (1), we get $$(4) \; \; \; \; \epsilon_n = n \cdot (H(K)-H(K|X_1)) = n \cdot \frac12 (\log_2 (n/(n-1)) + O(1/n)) = \\ \frac12 \log_2 ((n/(n-1))^n) + O(1) \rightarrow \log_2(\sqrt{e}) + O(1) \ .$$

By slightly improving the approximation (3) we should be able to replace the $O(1)$ term in (4) by $O(1/n)$ and therefore get that, indeed, $\lim_{n \rightarrow \infty} \epsilon_n = \log_2 \sqrt{e}$. To get this better estimation it should be enough to check that $H(Binom(n,1/2))=\frac12 \log_2 ( \pi en / 2) + err(n)$ where $err(n)=O(1/n)$ and $err$ is a monotone function.

• Very nice! I accept usul's answer since your answer is inspired by usul's, but your answer is very elegant (and converts to $\log(\sqrt e)$, too). – Danny Apr 10 '14 at 10:08
• *converges instead of "converts" :) – Danny Apr 10 '14 at 11:29
• Danny, I'm not sure it's a good idea to accept a more complicated and less elegant answer. Everyone else who ever looks at this question will look at @usul's answer first, while it's in everyone's interest that they look at the most simple and elegant answer first. The solutions to this are: 1. to ask usul to reformulate his answer to be simpler (for all I care, he could copy the text of my answer verbatim if he wants, I don't care about the reputation or anything) or 2. to accept my answer. I'll transfer the relevant reputation to usul if that's possible. – mobius dumpling Apr 10 '14 at 13:30
• This seems like the better answer to accept in my opinion too! Very nice/elegant entropy argument. – usul Apr 15 '14 at 18:10
• Note that in the literature you'll often see $\log_2\sqrt{e}$ written as $\frac{1}{2\ln 2}$. – Steven Stadnicki Apr 18 '14 at 18:52

(Edited from previous version, 2014-04-08.)

I believe that the answer is $\epsilon_n \to \log(\sqrt{e}) \approx 0.7213475...$ where the logarithm is base 2. This seems to match simulation results. I don't have a full formal proof, but give the heuristic approximations/calculations.

I think it's easier to note that your question is equivalent to:

What are the asymptotics of $\delta_n = 1 - \mathbb{E}[H(k/n)]$, where the expectation is over $k \sim Binomial(n,0.5)$?

(where $H$ is the binary entropy function $H(p) = p\log(1/p) + (1-p)\log(1/(1-p))$, and $\delta_n = \epsilon_n/n$.)

Here's a quick plot to show the idea. We have the binary entropy function in blue and the Binomial pmf (for $p=0.5$) in green. So we can see that the expectation of $H(k/n)$, when $k$ is distributed binomially, will always be below one but should be approaching one. The question is how fast.

The key idea will be that the value we're interested in, $$\mathbb{E}[H(k/n)] = \sum_{k=0}^n \frac{{n \choose k}}{2^n} H\left(\frac{k}{n}\right) ,$$ can be related entropy of the binomial distribution. Let $p_{n,k} = \frac{{n \choose k}}{2^n}$ be the probability of $k$ heads in $n$ fair coin flips. The steps will be as follows:

1. Use Stirling's approximation to get $H(k/n) = \frac{\log(p_{n,k})}{n} + 1 + \text{something}$.
2. Rewrite the original sum to get $\mathbb{E}[H(k/n)] = 1 - \frac{H(Binom(n,0.5))}{n} + \sum \text{something}$.
3. Get that the entropy of the binomial, divided by $n$, plus the sum of "something", equals $\frac{\log(\sqrt{e})}{n} + O(1/n^2)$.

Step 1:

Just plugging in Stirling and doing some cancellation/rearranging, \begin{align} p_{n,k} &:= \frac{{n \choose k}}{2^n} \\ &\sim \frac{n^n}{k^k (n-k)^{n-k}}\frac{1}{2^n}\sqrt{\frac{n}{2\pi k (n-k)}} . \end{align}

This won't be very tight for every term in the sum, but I think it will be asmyptotically tight towards the middle, which is all that matters since all the probability is in the center.

We can rewrite the entropy function as follows. It's just some arithmetic to combine the logarithms. \begin{align} H(k/n) &= \frac{k}{n}\log\left(\frac{1}{k/n}\right) + \frac{n-k}{n}\log\left(\frac{1}{(n-k)/n}\right) \\ &= \log\left(\frac{n}{k^{k/n}(n-k)^{(n-k)/n}}\right) . \end{align}

So, using Sterling's approximation above, the logarithm of a probability term is \begin{align} \log(p_{n,k}) &= \log\left(\frac{{n \choose k}}{2^n}\right) \\ &\approx n H(k/n) - n + \log\left(\sqrt{\frac{n}{2\pi k(n-k)}}\right) . \end{align}

Step 2: \begin{align} \mathbb{E}[H(k/n)] &= \sum_{k=0}^n p_{n,k} H(k/n) \\ &\approx \sum_{k=0}^n p_{n,k} \left(\frac{\log(p_{n,k})}{n} + 1 - \frac{\log\left(\sqrt{\frac{n}{2\pi k (n-k)}}\right)}{n}\right) \\ &= 1 - \frac{H(Binom(n,0.5))}{n} - \frac{1}{n}\sum_{k=0}^n p_{n,k}\log\left(\sqrt{\frac{n}{2\pi k(n-k)}}\right) . \end{align} Here, $H(Binom(n,0.5))$ is the entropy of the Binomial distribution for $n$ coin flips and $p=0.5$, which by wikipedia is $\log\left(\sqrt{\pi e n / 2}\right) + O(1/n)$.

Step 3:

Now, we just need to approximate the third sum. I will take a very rough approximation (feel free to do better, but it probably doesn't gain much). All the probability mass is concentrated on $k = \frac{n}{2} \pm o(n)$. So approximate this sum (which is an expectation) by its value on the term $k=\frac{n}{2}$, when it is $\log\left(\sqrt{\frac{2}{\pi n}}\right)$.

So now, we get \begin{align} \delta_n &\approx \frac{\log\left(\sqrt{\pi e n/2}\right)}{n} + \frac{\log\left(\sqrt{2/\pi n}\right)}{n} \pm O\left(\frac{1}{n^2}\right) \\ &= \frac{\log\left(\sqrt{e}\right)}{n} \pm O\left(\frac{1}{n^2}\right) . \end{align}

• Thank you very much for your effort. Your attempt is useful and i like your ideas, but i am not quite sure about the details. I will think about it and maybe i will ask some questions soon.. – Danny Apr 8 '14 at 21:35
• @Danny, sure thing. Let me know if parts are unclear. The computation is not fully rigorous, particularly (1) the use of Stirling's approximation everywhere (a fix might be to only consider an interval where the Binomial probability is $\Omega(1/n)$ and argue that Stirling is asymptotically tight there) and (2) the approximation of the "third term" by just setting $k=n/2$ (a fix might be to again consider the high-probability interval and see if the error is small). – usul Apr 9 '14 at 1:24
• My first question is a general question: You think that $\mathbb{E}_{w \in \{0,1\}^n} nH(w) = n-\mathcal{O}(1)$. Do you think there is a "short" proof for this conjecture without getting the exact term $\varepsilon_{n}$ respectively the exact limit of $\varepsilon_{n}$, which you are considering to be $\log\sqrt{e}$? – Danny Apr 9 '14 at 13:35
• @Danny, I'm not sure how to get this with a shorter proof, but maybe another answer will do so. – usul Apr 9 '14 at 20:51
• Thanks again for your help. Everything seems to be clear.. – Danny Apr 10 '14 at 9:07

Here is a proof that the quantity # that OP considers tends to n(1-o(1)).

Claim: #=n-o(n)

First, note that for any function $f:\{0, 1\}^n \rightarrow \mathbb{R}$, $\frac{1}{2^n} \sum_{w \in \{0,1\}^n} f(w)$ is exactly the same as $\mathbb{E}_{w \in \{0,1\}^n} f(w)$. So you're asking whether $\mathbb{E}_{w \in \{0,1\}^n} H(w) = 1-o(1)$.

To see this, define $X$ to be a random variable uniformly distributed on $\{0,1\}^n$ Let $X_0$ be the number of 0's in $X$ and $X_1$ be the number of 1's in $X$. We want to prove $$\mathbb{E}[(X_0/n)\log(n/X_0)+(X_1/n)\log(n/X_1)]=1-o(1).$$ But we know from the law of large numbers that both $X_0/n$ and $X_1/n$ converge in probability to $1/2$ and that if a random variable $Y$ converges in probability to $c$ and if $f$ is a continuous function, then $f(Y)$ converges in probability to $f(c)$. So we get that $\mathbb{E}[(X_0/n)\log(n/X_0)+(X_1/n)\log(n/X_1)]$ converges in probability to $\frac12 \log(2) + \frac12 \log(2) = 1$, QED.

Note: This claim is related to the Asymptotic equipartition property for discrete-time i.i.d. sources here.

• I'm sorry, but I can't see how $\mathbb{E}_{w\in\left\{0,1\right\}^{n}}\left[H\left(\frac{|w|_{0}}{n},\frac{|w|_{1}}{n}\right)\right]=1-o(1)$ is a direct result of the AEP, since we are looking for the expectation of entropies for all empirical distributions and the AEP treats with the limit of the selfinformations of strings for some fixed distribution. Can you specify that implication? – Danny Apr 8 '14 at 15:06
• Also I do not see why your statement $\mathbb{E}[(X_0/n)\log(n/X_0)+(X_1/n)\log(n/X_1)]=1-o(1)$ leads directly to my conjecture.. Thanks for your help. – Danny Apr 8 '14 at 15:18
• So you're asking whether $\mathbb{E}_{w\in\{0,1\}^n}H(w) = 1 - o(1)$. Note this isn't quite so, the question as worded is whether it is $1 - o(1/n)$. – usul Apr 9 '14 at 1:26
• @Danny, the expression $\mathbb{E}[(X_0/n)\log (n/X_0) + (X_1/n) \log (n/X_1)]$ is exactly the same as your expression $H_0(w)$ (or, in my notation, $H_0(X)$). – mobius dumpling Apr 9 '14 at 11:39
• @Danny You're right, the theorem does not follow from the AEP as easily as I thought. Rather than explain how it follows, I'll just leave my self-contained explanation, which turns out to be simpler than relating to the AEP. – mobius dumpling Apr 9 '14 at 11:44

I will prove that for all $n$, $H_n = n - \epsilon_n$ where $0 < \epsilon_n < 1$.

Let $f_{\alpha}(t) = 4^{\alpha} t^{\alpha}(1 - t)^{\alpha}$. I will not prove this, but you can find some $\alpha^*$ such that for all $\alpha \ge \alpha^*$, $f_{\alpha}(t) \le H(t)$. Also, for $\delta \ll 1$, you can find $\alpha'$ such that for all $\alpha \le \alpha'$, $f_{\alpha'}(t) \ge H(t)$ in $[\delta, 1 - \delta]$. I will assume $\alpha = 1$ and $\alpha = 0.5$ lead to the two different types of functions.

My point is that you can use simpler functions to obtain the bound you need. Here I will use $f_{\alpha}(t)$.

Let $H_n$ be the empirical entropy that you are looking for and $F(n, \alpha)$ be the value obtained when I replace $H(\frac{k}{n})$ with $f_{\alpha}(\frac{k}{n})$. We have

$F(n, \alpha) = n2^n \sum_{k = 0}^{n} {n \choose k} f_{\alpha}(\frac{k}{n}) = n 2^n 4^{\alpha} \sum_{k = 0}^{n} {n \choose k} \left( \frac{k}{n} \right) ^ {\alpha} \left(1 - \frac{k}{n} \right) ^\alpha = n 4^{\alpha} E\left[\left( \frac{k}{n} - \frac{k^2}{n^2} \right)^{\alpha} \right] = n2^n 4^{\alpha} \sum_{k = 0}^{n / 2} {n \choose k} \left(\frac{k}{n} \right)^{2 \alpha}$

For $\alpha = 1$, we know $F(n, \alpha) \le H_n$ and direct evaluation says that $F(n, 1) = n - 1$, which is true for all $n$.

For $\alpha = 0.5$, we know $F(n, \alpha) \ge H_n$ and a direct evaluation leads to $F(n, 0.5) = n 4^{0.5} E[\frac{k}{n}] = n$.

Therefore, for all $n > 0$, $n\ge H_n \ge n - 1$.

I found it nontrivial to try this for any $\alpha < 1.0$ other than $\alpha = 0.5$.


Coming up with a formula for

1. Sep 12, 2004

Caldus

Um, how can I figure out a formula for solving something like:

1 + 3 + 5 + 7 + 9 + ... + n

In particular: 1 + 3 + 5 + ... + 999

Thanks.

2. Sep 12, 2004

geometer

As far as I know, there is no rote procedure for doing this. It's pretty much a trial and error process. Make yourself a table and figure out the pattern. In this case, we can write this as the sum from x = 0 to x = (n-1)/2 of 2x+1:

x = 0 ===> Sum = 1
x = 1 ===> Sum = 4
x = 2 ===> Sum = 9
x = 3 ===> Sum = 16

We can keep this up if we need to to help us see the pattern, but in this case, a little thought shows that the formula is: Sum from x= 0 to x = n = (n+1)^2

3. Sep 12, 2004

Integral

Staff Emeritus
I believe that Gauss did this as school child when given the assignment to sum the digits 1-100.

First note that if you add them together smallest to largest you get a constant result ( 100+ 1=101, 99+2=101 ..) multiply this by 100 because there are 100 sums and divide by 2 because you added all the numbers twice.

edit:
OPPS! just noticed that you were not summing ALL integers.

Last edited: Sep 12, 2004
4. Sep 12, 2004

Integral

Staff Emeritus
After a bit more playing,

consider this

$$S = \Sigma _ {i=1}^N (2i -1)$$

now using the properties of the summation

$$S= 2\Sigma_{i=1}^N i - \Sigma_{i=1}^N 1$$

$$S= 2\Sigma_{i=1}^N i -N$$

Now using Gauss's trick we can see that the sum of n integers is given by

$$S_n = \frac {(n+1)(n)} 2$$

So we can write for the sum of the odd integers

$$S = (N+1)(N)-N = N^2$$

Note that I have given the general expression for the odd integers as 2n-1 and my sum starts with n=1 where geometer used 2n+1 and started with n=0. This is the reason he has N+1 where I have N. Also note that this N is NOT the odd integer you are summing to but the NUMBER OF TERMS in the sum.

Last edited: Sep 12, 2004
5. Sep 12, 2004

shmoe

Integral, the same idea of pairing highest and lowest will work just fine,
1+3+5+7+9+11=12+12+12=(6/2)*12, where 6 is the number of terms here.

If you already have a formula for the sum of the first n integers, 1+2+3+..+n, you can use this as well:
1+3+5+7+9+11=(1+2+..+11)-(2+4+6+8+10)=(1+2+..+11)-2(1+2+3+4+5)=(insert formula here)-2(insert formula here)

Caldus, do you have a formula for the sum of the first n integers? If not, see Integrals post;)

Another idea, (which Integral provided while I was typing) write your sum as: $$\displaystyle\sum\limits_{i=1}^{n}(2i-1)$$ and fiddle with it to get an expression involving $$\displaystyle\sum\limits_{i=1}^{n}i$$, and apply the formula for this (again, using the ideas in Integral's post if need be). This would be a more easily applied general idea that can handle any arithmetic series $$\displaystyle\sum\limits_{i=0}^{n}(ki+d)$$.

Last edited: Sep 12, 2004
6. Sep 13, 2004

HallsofIvy

Staff Emeritus
Since you are adding only odd numbers, one way to do is this (Assuming you know that the sum of the first n integers is n(n+1)/2):
The sum of all numbers from 1 to 2n is (2n)(2n+1)/2= n(2n+1). Now subtract off the sum of all even numbers from 2 to 2n: 2+ 4+ 6+ ...+ 2n= 2(1+ 2+ 3+ ... + n)=
2(n)(n+1)/2= n(n+1).

The sum of all odd numbers from 1 to 2n-1 is n(2n+1)- n(n+1)= n(2n+1- n- 1)= n2 just as shmoe said.

In particular, the sum of all odd numbers from 1 to 99 is: (99= 100- 1= 2(50)- 1)
502= 2500.

7. Sep 13, 2004

Gokul43201

Staff Emeritus
Also, here's a general rule that can be used in times of trouble :

If a series S, has terms t(n), which are polynomials in n, of order p, the sum S(n) is a polynomial of order p+1. The coefficients of this polynomial can be determined simultaneously, from the given p+2 terms in the series.

Example : If the series S, has terms, t(n) which are linear in n, ie : t(n) = an + b, then the sum, S(n) is a quadratic in n, ie: S(n) = An^2 + Bn + C

NOTE : The order of the polynomial can be determined by the method of "successive differences". If the p'th successive differences are equal, the order is p.

Example : Consider the sequence : 2, 6, 12, 20, 30, ...
The first differences are : 4, 6, 8, 10, ...
The second differences are : 2, 2, 2, ...

Since the second differences are equal, t(n) is a quadratic in n, ie : t(n) = an^2 + bn + c.

So you can assume the sum has a cubic form : S(n) = An^3 + Bn^2 + Cn + D
Plugging in the for S(1), S(2), S(3) and S(4) gives you 4 simultaneous equations in A, B, C, and D, which you can solve.


# How can we find the number of pairs of intersecting ranges on a circular number line?

I recently thought of and managed to solve this algorithmic problem:

On a infinite 1-dimensional number line, we have N ranges specified by two distinct integers A and B, such that all A and B are unique. Each pair of integers then represents the starting and ending points of a range on the number line. How many distinct pairs of ranges intersect?

^^ In this example, 4 distinct pairs of segments intersect.

Obviously, there is a O(N^2) brute-force solution: just test all possible pairs in O(1) time each. However, I also thought of a O(NlogN) sweep-line algorithm: treat every A as a "segment start event" and every B as a "segment end event". Then, keep a variable curr_in_range (originally 0) that stores the current number of segments that we currently see, and then iterate through the events in sorted order of their positions on the number line. Every time we add another segment, add curr_in_range to the answer (because this new segment will intersect with every other segment that we can see) and increment it as well. Every time we remove a segment, decrement curr_in_range. In the end, this gets us the correct answer.

Happy with this algorithm, I decided to tackle a slightly modified version of this problem: What if the number line is circular? As in, it has a specified length L such that the number line contains numbers in the range [0, L)? In this case, each range would still be specified by two distinct integers A and B, but let's just say the range is always counterclockwise starting from A and ending at B. Could we still find the number of pairs of sectors that intersect in less than O(N^2) time in this scenario?

Here's my thought process on this problem:

We should be able to use a very similar algorithm to the one described above. Completely copying it wouldn't work, though, because the sectors can have B < A when it crosses over 0. My idea was to initially have curr_in_range equal to the number of sectors that cross over 0 (instead of just it at 0 like we did in the above algorithm). Then, we could treat every sector as a "sector start event" and a "sector end event", and do the same process that we did.

However, after extensive analysis, I found why this approach doesn't work. This approach won't work because it overcounts some of the intersections, as sometimes two sectors can intersect in 2 places. This is where I'm stuck.

^^ Our sweep-line algorithm will count 2 distinct pairs that intersect, when there is actually only 1.

But besides this overcounting, I found that this current algorithm isn't doing a single other thing wrong. I coded a O(N^2) brute-force algorithm and tabulated all the distinct pairs that it found, and also tabulated the distinct pairs found for the current algorithm. For every randomly-generated test case, these two tables were the same. In other words, our current algorithm is correctly detecting every intersection there is to be detected, but it just happens to overcount some of them.

So, sorry for a really long post, but my final question is:

Can someone come up with an algorithm that solves this problem? Is my thought process close to a solution or completely off?

• Simple idea that might work: sort intervals as usual, except give each one a unique id. Begin processing using the same algorithm, except every start interval is marked as started, and if you encounter an end interval that hasn't been marked, you just ignore it. The algorithm terminates when all intervals are marked as started. – hLk Mar 27 at 4:22

This is solvable in $$O(n \log n)$$ time, too, by considering the complement of the intervals that wrap around and using a suitable data structure.

There are two types of intervals that can appear on a cycle:

• Intervals that do not wrap around: e.g., $$[0.3,0.5]$$ does not wrap around. In general, this includes all intervals of the form $$[a,b]$$ with $$0\le a \le b \le 1$$.

• Intervals that wrap around: e.g., $$[0.8,0.1]$$ wraps around; it is equivalent to $$[0.8,1.1]$$ or $$[0.8,1] \cup [0,0.1]$$. In general, this includes all intervals of the form $$[b,a]$$ with $$0\le a < b \le 1$$; each such interval is equivalent to $$[b,1] \cup [0,a]$$.

Let $$A$$ denote the set of intervals that do not wrap around, and $$B$$ the set of intervals that do wrap around. Also, let $$\overline{B}$$ denote the complements of the intervals in $$B$$, i.e., $$\overline{B} = \{\overline{b} \mid b \in B\}$$. For instance, the interval $$[0.8,0.1]$$ wraps around; its complement is $$(0.1,0.8)$$, which does not wrap around. It is easier to think about intervals that do not wrap around, and this decomposition will help us to deal only with intervals that do not wrap around.

Now, we will separately count the number of overlaps between (1) an interval in $$A$$ and an interval in $$A$$; (2) an interval in $$A$$ and an interval in $$B$$; (3) an interval in $$B$$ and an interval in $$A$$; and (4) an interval in $$B$$ and an interval in $$B$$. (1) can be counted with your sweepline algorithm, as we don't need to deal with wraparound. (4) can also be counted in the same way, as two intervals overlap iff their complements overlap. I will show below how to count (2); case (3) will then follow by symmetry. This covers all the cases, so it suffices to count up these four cases and sum them up.

Note that intervals $$a \in A, b\in B$$ overlap iff $$a$$ is not a subset of $$\overline{b}$$. So, it suffices to count the number of pairs $$a \in A, \overline{b}\in \overline{B}$$ where $$a$$ is a subset of $$\overline{b}$$; then you can subtract that from $$|A| \cdot |B|$$.

To do that, you can use the data structure in Data structure for interval subset queries. Store all the intervals $$\overline{B}$$ in the data structure. Then, for each $$a \in A$$, count the number of intervals in $$\overline{B}$$ that $$a$$ is a subset of. This can be done in $$O(\log n)$$ time per query interval $$a$$, for a total time of $$O(n \log n)$$.

Therefore, you can count each of the four cases in $$O(n \log n)$$ time, and thus compute the overall sum in $$O(n \log n)$$ time as well. I think. I encourage you to check my reasoning to make sure I haven't made any mistakes.

• "(4) can also be counted in the same way, as two intervals overlap iff their complements overlap." A range overlaps any wrapping range that contains the whole circle, but their complements do not overlap. Anyway, in case (4) where all ranges are wrapping, we can just say any two of them overlap. – John L. Mar 25 at 18:49
• Apparently, I could not understand how your usage of "a segment tree to obtain that count" avoids counting the same intersecting pair more than once. – John L. Mar 25 at 18:54
• @JohnL., By "wrapping" I mean a range like $[0.9,0.1]$, i.e., $[0.9,1.1]$, i.e., $[0.9,1.0] \cup [0,0.1]$. Such a range does not cover the whole circle, but includes $1$ and $0$ in the range. The only range that covers the entire circle is $[0,1]$, and that range is non-wrapping. – D.W. Mar 25 at 18:58
• Are you talking about 2-dimensional segment trees? What are the leaves of your segment tree? If each leaf is an elementary interval, how can each interval of $\overline B$ appear in the segment tree only once? If a leaf can be an interval of $\overline B$, then how can an interval of $A$ be expressed as concatenation of those leaves? – John L. Mar 25 at 19:18
• @JohnL., No, an ordinary segment tree for storing the intervals of $\overline{B}$. As usual, we take the endpoints of the intervals of $\overline{B}$, sort them, obtain the elementary intervals, and each leaf corresponds to one elementary interval (not an interval of $\overline{B}$). Each interval of $\overline{B}$ is stored in some node of the segment tree, following the rule described at the link you provided. The interval $a$ can be expressed as a disjoint union of elementary intervals, not of intervals of $\overline{B}$. – D.W. Mar 25 at 19:22

Not a solution, just an illustration of what I see as the core difficulty:

The instance on the left has 3 intersections, the one on the right only 2, but there is no way to distinguish the instances just by counting numbers of "begin segment" and "end segment" events within a given interval. (Inside the gap left by the red interval, the event sequence is "begin", "end" in both cases.)

This implies that a sweep-line approach needs to track more information about each active segment -- probably their identities, which suggests to me that no fast algorithm will be possible, though that's just a feeling.

• A sweep algorithm will take $\Omega(n\log(n))$ time anyways since it has to sort the array. Then, we might be able to store the identities in a way it would not take much more space and time, so we would not overshoot that $n\log(n)$. Not proposing a solution, but just saying that even if we have to keep the identities it might be fast enough – nir shahar Mar 12 at 23:33

### Summary

Interesting question.

Please check this program in Java that runs in $$O(n\log n)$$ time, where $$n$$ is the number of given ranges. The program uses sweep-line technique, once for counting intersecting pairs of ranges, and once for counting containing pairs of ranges together with Fenwick tree. Explanations follow.

### Reduce to the hard case

Here are the notations and assumptions for the ease of explanation.

• All numbers and variables are integers.
• The numbered circle of cardinality $$L\ge2$$ is the set of number $$0, 1, \cdots, L-1$$, where $$L$$ is considered to wrap around to $$0$$.
• A range $$[s,e]$$ with respect to $$L$$, where $$0\le s\le e\lt L$$, means
• a non-wrapping range (a.k.a. an ordinary range) if $$s\le e$$, i.e., numbers $$s, s+1,\cdots, e-1, e$$;
• a wrapping range otherwise, i.e,. numbers $$s, s+1, \cdots, L-1, 0, 1, \cdots, e-1, e$$.

Note every range contains at least one number. The non-wrapping range $$[0, L-1]$$ contains all the numbers. So does every wrapping range like $$[s,s-1]$$.

Suppose there are $$n$$ ranges. The question is how to compute the number of all unordered pairs of two intersecting ranges efficiently.

There are three cases for an unordered pair of two ranges.

• Both ranges are wrapping.
This is the easiest case since two wrapping ranges always intersect at $$0$$.

• Both ranges are non-wrapping.
This is the classical case. The total number of such intersecting pairs can be computed by the sweep-line (or, as I would call it, the sweep-point or sweep-event) algorithm, which runs in $$O(n)$$ time, as described in the question.

• One range is non-wrapping and the other is wrapping.
This is the new and hard case. One way to understand why this case may not be easy is j_random_hacker's illustration. How can we deal with this case?

### Transform to the containing pairs of non-wrapping pairs

Denote the non-wrapping ranges and the wrapping ranges in the given ranges by $$\mathcal N$$ and $$\mathcal W$$, respectively. Instead of counting the intersecting pairs between $$\mathcal N$$ and $$\mathcal W$$, we will use complementary counting, i.e., we will count the non-intersecting pairs between $$\mathcal N$$ and $$\mathcal W$$.

Given any range $$r$$, let $$\overline r$$ be the numbers among $$0, 1, \cdots, L-1$$ that are not in $$r$$, i.e., the complement of $$r$$ with respect to $$\mathcal C$$. Note that two ranges do NOT intersect iff the complement of either one contains the other.

Let $$\overline{\mathcal W} = \{\overline r\mid r\in\mathcal W\}$$. Then the number of non-intersecting pairs between $$\mathcal N$$ and $$\mathcal W$$ is the number of containing pairs between $$\mathcal N$$ and $$\overline{\mathcal W}$$. Here a containing pair means, naturally, two sets $$a$$ and $$c$$ such that $$c$$ contains $$a$$. The number we wanted, i.e., the number of all intersecting pairs of ranges between $$\mathcal N$$ and $$\mathcal W$$ is $$|\mathcal N|\cdot|\mathcal W|$$ minus that number of containing pairs.

More formally, we have $$\#\{(a, b) \mid a\in{\mathcal N}, b\in{\mathcal W}, a\cap b\not=\emptyset\} = \#{\mathcal N}\cdot\#{\mathcal W} - \#\{(a, c) \mid a\in{\mathcal N}, c\in\overline{\mathcal W}, a\subseteq c\}.$$

Here is a simple observation, the complement of a wrapping range is a non-wrapping range, except when the wrapping range is like $$[s,s-1]$$, whose complement is empty. Containing all numbers, a wrapping range like that intersects with every other range. It is easy to count all intersecting pairs that involve a wrapping range like that.

Let us assume there is no such wrapping range; if necessary, we can replace such a range with non-wrapping range $$[0, L-1]$$. The problem is reduced to counting the number of containing pairs between two sets of non-wrapping ranges.

### An algorithm that counts the number of containing pairs

Here is an algorithm that counts the number of containing pairs between two given sets of non-wrapping ranges, $$\mathcal N$$ and $$\overline{\mathcal W}$$. It is, basically, sweep-line technique together with Fenwick tree.

1. Let containingPairCount = 0.
2. Initialize a Fenwick tree (a.k.a. binary indexed tree) ft for an (implicit) interested array of length $$L$$, with all values 0 initially. The e-th element of the interested array will be the number of all ranges in $$\overline{\mathcal W}$$ so far that end at e.
3. Sort all ranges that are in either $$\overline{\mathcal W}$$ or $$\mathcal N$$ by the starting points, breaking ties by putting ranges in $$\overline{\mathcal W}$$ before ranges in $$\mathcal N$$. Breaking the remaining ties arbitrarily.
4. One by one process the sorted ranges. Let the current range be [s,e].
• If it comes from $$\overline{\mathcal W}$$, updates the Fenwick tree as if the e-th element of the interested array is increased by 1. This operation registers the current range as a candidate containing range that has started and will end at e.
• If it comes from $$\mathcal N$$, add the sum of elements of indexes no smaller than e in the interested array to containingPairCount, by containingPairCount += st.getSum(e, L-1).
5. Return containingPairCount.

Proof of the algorithm: st.getSum(e, L-1) is the number of all ranges in $$\overline{\mathcal W}$$ that contain the current non-wrapping range, since, thanks to the careful setup, we have processed all ranges in $$\overline{\mathcal W}$$ that start at or before the start of the current range, and only those ranges that end at or after the end of the current edge are counted. $$\quad\checkmark$$

The sorting of all ranges at step 3 takes $$O(n\log n)$$ time. At step 4, each range is processed with $$O(\log n)$$ time, thanks to the power of Fenwick tree. So the running time of this algorithm is $$O(n\log n) + O(n \log n) = O(n\log n)$$.

### Complexity analysis

The running time for other parts of the entire algorithm is $$O(n\log n)$$, which is spent mostly on counting the intersecting non-wrapping pairs. So, the running time of the entire algorithm is $$O(n\log n) + O(n \log n) = O(n\log n)$$.

The sections "Reduce to the hard case" and "transform to the containing pairs of non-wrapping pairs" follow D.W's answer roughly.