Chapter 3: Linear systems and row reduction
3.6Proof patterns
Nearly every proof in this chapter ran through Gauss–Jordan elimination, but it used the algorithm in only a handful of ways: as a reduction to a simpler matrix, as an object whose correctness had to be proved, as a way to settle existence questions, and as a table from which structure can be read. This section sorts the chapter’s proofs by the question they answer, compares the tools on examples from the chapter, and says when each tool is the wrong one. The exercises at the end practice one move at a time, and the chapter’s problems (Problems for Chapter 3) mix them.
3.6.1Proving a statement about every matrix
The chapter has two reductions. Technique 3.3.22 replaces by with invertible (Theorem 3.3.10). Technique 5.4.6, which is named in Section 5.4 but first used here, writes an invertible matrix as a product of elementary matrices (Theorem 3.5.19) and checks the three types one at a time. The rank normal form (Theorem 3.5.27) is a two-sided version of the first. To choose between them, ask what the property you want has to survive.
Reducing to RREF needs a property with for every invertible . The chapter found many: the kernel (Corollary 3.2.23), the linear relations among the columns (Corollary 3.2.17), and everything defined from these, such as which columns are pivot columns (Corollary 3.3.19), the rank (Proposition 3.5.2(c)), and, for square , invertibility. Solution sets survive only if the right-hand side moves too: becomes (Proposition 3.2.14). Once invariance is checked, the question is about a matrix whose pivot columns are and whose zero rows are visible, and it usually answers itself. This is how Theorem 3.3.23, Proposition 3.4.15, and Corollary 3.3.20 were proved.
Reducing to elementary matrices needs something else: a statement about invertible matrices that survives products. Then two checks suffice, one computation for each type and one step showing that the statement passes from to (or ). Invariance under plays no role. This is how Corollary 3.5.22 turned “multiply by invertible matrices” into “perform row and column operations”, and how Section 5.4 proves
Example 3.6.1 (Which reduction fits). Commuting with every invertible matrix. Which satisfy for every invertible ? Reducing to RREF is useless here, since commuting with says nothing about commuting with . But the property survives products: if and , then
Rank under row and column operations. That for invertible is one line by reducing to RREF:
3.6.2Proving that an algorithm is correct
An algorithm proves that something exists once two things are shown about every run (Technique 3.3.5). For termination, find a quantity that moves in one direction and cannot do so forever: in Gauss–Jordan elimination, the number of pivoted rows grows by in each pass and never exceeds (Proposition 3.3.6). For correctness, find an invariant that the starting state satisfies and every step preserves, and that, combined with the reason for halting, gives the property of the output.
The invariant is where the work is, and its typical feature is visible in Lemma 3.3.7. The property we want at the end, conditions (R1)–(R4), is not preserved pass by pass: halfway through, the unpivoted rows are an arbitrary mess. The invariant has to say something about that mess, and part (e) does: the unpivoted rows vanish in every column up to the last pivot. Part (e) is what forces each new pivot to lie to the right of the old ones, which is (b), and it, together with the vanishing of the new pivot row to the left of its pivot, is exactly what choosing the leftmost column in step 1 of Algorithm 3.3.1 keeps true. Drop that choice and both the proof and the algorithm break (Exercise 3.3.47). The same pattern recurs in the exercises: Euclid’s algorithm by row operations has the monovariant and the invariant “the common divisors of the two entries” (Exercise 3.2.54), and elimination on a strictly diagonally dominant matrix needs no swap because the block still to be processed stays diagonally dominant (Exercise 3.3.56). Even Theorem 3.3.12 was proved by running the algorithm, on a matrix of the right shape, with the invariant “nothing has changed yet”. Exercise 3.6.11 practices the technique on an algorithm that has nothing to do with elimination.
An algorithm proves that some output with the guaranteed properties exists. It does not prove that this output is the only object with those properties, and a run that made different choices could, as far as the invariant knows, stop somewhere else. That is why the uniqueness of the reduced row echelon form needed its own theorem (Theorem 3.3.17), and why, before it, we could not say that another way of reducing finds the same pivots, or that (Remark 3.3.2). For echelon forms that are not reduced, uniqueness really fails (Example 3.3.29).
3.6.3Turning an existence question into a linear system
“Is there a polynomial, a combination, a map, a set of weights, such that …?” The move of Technique 3.1.9 is to write the unknown object with unknown scalars, translate each requirement into an equation in them, and check that the equations are linear. The question has become the consistency of a linear system, and the set of all answers has the shape of Technique 3.1.18: one answer plus everything that solves the homogeneous version. The chapter used this for membership in a span (Example 3.1.10), interpolation (Theorem 3.4.24), chemical balancing (Exercise 3.1.39), quadrature weights (Exercise 3.4.43), and antiderivatives, where the homogeneous solutions are the constants (Example 3.1.20).
Example 3.6.2 (One question, three answers). Which polynomials take the values at ? The answer depends on how many coefficients we allow, that is, on the number of unknowns against the three equations (Exercise 3.1.37).
- In , write . The conditions , , give and , so there is no solution: three equations, two unknowns, and an inconsistent system.
- In there are three unknowns and exactly one solution, .
- In there are four unknowns, and the solutions are
for : the particular solution from the previous case plus the cubics that vanish at .
The count of unknowns against equations predicts what can happen (Corollary 3.4.19), but only the system decides. The square case has exactly one solution here because of a structural fact, which is the subject of the next subsection.
The technique is the wrong tool when the requirements are not linear in the unknowns. Asking for a matrix with gives quadratic equations in the entries of , and elimination does not apply. Side conditions such as positivity or integrality are not linear either: the linear system only finds the subspace of all balancings in Exercise 3.1.39, and a separate divisibility argument picks the smallest positive integer one; the nonnegative flows of Exercise 3.1.38 come from an inequality on the parameter. Also remember that the system lives over a field, and solving it divides by scalars: in Example 3.1.10 the same equations are solvable over and inconsistent over , because one step divided by .
3.6.4When uniqueness gives existence
Uniqueness and existence behave differently. Uniqueness is a property of the homogeneous problem alone, for a matrix of any size: a consistent system has exactly one solution if and only if (Corollary 3.1.22). Existence for every right-hand side is a property of the zero rows of (Proposition 3.4.16). For a square matrix the two conditions coincide, since both say (Corollary 3.4.20). So for square problems the homogeneous statement buys existence for free (Technique 3.4.21). The homogeneous statement says that a single object is zero, and it is usually proved by structure rather than computation: a polynomial with too many roots (Lemma 3.4.23), a maximum principle (Exercise 3.1.47), or the observation that leaves no room for free variables (Theorem 3.5.23).
There are three ways to misuse the technique. First, on a non-square system it proves nothing: has trivial kernel and has no solution (Exercise 3.1.34). Second, counting conditions is not the proof; the homogeneous problem must actually have only the zero solution, and when it does not, existence fails too for some data. In the cyclic system of Exercise 3.1.46 with even, the homogeneous solution comes with the condition that the alternating sum of the right-hand sides vanish, since the alternating sum of the left sides is . Exercise 3.6.14 is a smaller instance. Third, the technique depends on finite size. On , the shift with has a left inverse but is not surjective (the intuition after Theorem 3.5.23), and Section 4.2 replaces “square” by .
3.6.5Reading structure off the reduced form
Once is computed, a great deal can be read off without further work:
- a pivot in every column means at most one solution for each , independent columns, and injectivity of (Proposition 3.4.15, Theorem 3.5.12); a pivot in every row means solutions for every , spanning columns, and surjectivity (Proposition 3.4.16, Theorem 3.5.13);
- the pivot columns are the columns of that are not combinations of earlier ones, and the entries of in a free column are the coefficients that build that column of from the pivot columns (Corollary 3.3.19);
- the basic solutions, one for each free column, describe (Theorem 3.4.9);
- the last rows of give the consistency conditions, so they describe (Corollary 3.4.4), and they are also the relations among the rows of (Exercise 3.3.57).
Everything in the first three items is about relations among the columns, and carries it alone. The image is different: it needs , or the original columns of .
Example 3.6.4 (Everything about one matrix). For the matrix of Example 3.3.3, with columns ,
The rank is , with pivot columns and free columns . From the entries of , and . The basic solutions and describe (Example 3.4.10). The last row of gives the single consistency condition , so is the plane (Example 3.4.6), spanned by the pivot columns and of .
The classic error is to take the pivot columns of instead. They are and , and is not in , since . The columns of span , which is moved by (Non-example 3.2.19). The fix is to read which columns are pivot columns from and to take those columns from . Exercise 3.6.6 and Exercise 3.6.7 practice the distinction.
3.6.6Row operations and column operations
Row operations multiply on the left by an invertible , and column operations multiply on the right by an invertible . They preserve different things (Table 3.6.1).
| Property of | Row operations, | Column operations, |
|---|---|---|
| kept (Corollary 3.2.23) | moved by | |
| relations among the columns | kept (Corollary 3.2.17) | changed |
| , the span of the columns | moved by | kept (Proposition 3.2.30) |
| relations among the rows | changed | kept (Exercise 3.6.8) |
| and the pivot columns | kept (Corollary 3.3.18) | changed |
| kept (Proposition 3.5.2) | kept (Exercise 3.5.41) |
So row operations are the tool for solving systems and for finding relations among the columns, and column operations are the tool for questions about the span of the columns. Using column operations to solve changes the unknowns: is solved by exactly when is solved by , so the answer must be translated back. Using row operations on a question about the span of the columns moves the span. This is why the bad right-hand side in the proof of Proposition 3.4.16 is and not (Remark 3.4.17). With both kinds of operations allowed, only the rank survives, and every matrix of rank becomes (Theorem 3.5.27).
3.6.7Reading the chapter’s proofs
Uniqueness of the reduced form, two ways. Writing column by column suggests the first proof of Theorem 3.3.17: track , and show pivot by pivot, from the left, that the pivot columns agree and that fixes . It needs invertibility of in a concrete form, the independence of its columns, to rule out a pivot that appears too early in one matrix. The second proof forgets . It notices that everything about a matrix in RREF, which columns are pivot columns and what the other columns are, is determined by the linear relations among its columns (Lemma 3.3.16), and that row operations preserve those relations. The comparison teaches a habit: after a proof, list the hypotheses it actually used. The second proof used one consequence of , namely that and have the same kernel, so it proves the stronger statement Corollary 3.3.20 at no extra cost. The first proof, in turn, says something the second does not: is rigid on and free elsewhere (Exercise 3.3.46).
Constructive and structural proofs. Several results got one proof of each kind. Proposition 3.4.15 was proved by setting one free variable to , which produces a nonzero solution, and by the linear dependence lemma, which explains what a missing pivot means. Proposition 3.5.9 was proved from the pivots, which gives a formula, and by extending the columns to a basis, which works for linear maps between abstract vector spaces as long as the domain has a finite basis and the target a finite spanning list (Exercise 3.5.40). Theorem 3.5.27 was proved by clearing columns and by row reducing the transpose. The computational proof usually produces a formula or a witness, and the structural one usually survives the passage to abstract vector spaces.
Do it to the identity. To find the matrix of an operation that acts on each column by the same linear map, perform the operation on (Lemma 3.2.6, the first proof of Proposition 3.2.7). The same idea, run along with the algorithm, records the whole elimination: carrying next to produces (Proposition 3.3.11), and carrying next to a square produces (Algorithm 3.3.24).
3.6.8Common errors
Each of these appears in the chapter, most of them as find-the-error exercises.
- Simultaneous row operations. Replacing two equations at once, where the second replacement uses an equation the first one changed, can gain solutions (Exercise 3.1.35). So can scaling a row by , which is why it is not an elementary operation.
- The order of the factors. The first row operation is the rightmost factor of (Example 3.2.8); for column operations it is the leftmost (Remark 3.2.29).
- Row operations do not preserve the image. is equivalent to , not to (Exercise 3.2.43, Remark 3.4.17), and the columns of do not span (Exercise 3.6.6).
- Uniqueness is not existence. A trivial kernel gives at most one solution, not one (Exercise 3.1.34). Checking that some vectors are solutions does not show that they are all of them (Exercise 3.4.34, Remark 3.4.12). Counting equations and unknowns does not replace proving uniqueness (Exercise 3.6.14).
- The shape bounds the rank only from above. , and equality is a condition to check, not a consequence of the shape (Problem 3.7).
- One-sided inverses of non-square matrices. Cancelling a matrix that has only a right inverse, or inverting when is not square, is invalid (Exercise 3.5.26, Exercise 3.5.35).
- Rigidity overstated. with in RREF does not force (Exercise 3.3.46).
- Reducing to elementary matrices beyond its reach. Only invertible matrices are products of elementary matrices; the other case needs its own argument (Exercise 5.4.35, Problem 3.7).
- Hidden division. Pivots, ranks, and inverses depend on the field, because elimination divides (Example 3.5.3, Exercise 3.3.27).
- Pictures from the plane. Three planes in , no two of them parallel, can still have no common point (Exercise 3.1.36).
Exercises
Exercise 3.6.6. Let
A student argues: “Row operations do not change the linear relations among the columns, and the pivot columns of are independent. So has the independent spanning list , the pivot columns of .”
- (a)Show that , and find the error.
- (b)Find two vectors that span , and an equation for .
Uses. Corollary 3.3.19 (Pivot columns, intrinsically), Corollary 3.2.17 (Row operations preserve relations among columns), Proposition 3.1.8 (Systems and the map ), Corollary 3.4.4 (Consistency), Non-example 3.2.19 (Row operations move the column space)
Techniques. Reduce to RREF (Technique 3.3.22)
(a) The columns of are , , and , and each satisfies . This condition is preserved by sums and scalar multiples, so every linear combination of the columns satisfies it, and by Proposition 3.1.8(a) every vector of does. The vector has , so it is not in .
The first sentence of the argument is correct (Corollary 3.2.17), and it does show that columns and of are independent and that column is a combination of them. The error is the step from relations to the vectors themselves. Writing with invertible, the columns of are , so they span , the image of moved by . Row operations move the image (Non-example 3.2.19).
(b) By Corollary 3.3.19(b), the non-pivot column is a combination of the pivot column (indeed , read off the entry of ). So
so . Hence
Exercise 3.6.7. A matrix has
and its first and third columns are and .
- (a)Find .
- (b)Find vectors spanning , and an equation describing .
Uses. Corollary 3.3.19 (Pivot columns, intrinsically), Theorem 3.4.9 (All solutions), Definition 3.4.8 (Basic solutions), Proposition 3.1.8 (Systems and the map )
Techniques. Reduce to RREF (Technique 3.3.22)
(a) The pivot columns of are and , so these are the pivot columns of , and columns and are free. By Corollary 3.3.19(b), each free column of is the combination of the pivot columns to its left whose coefficients are the entries of the corresponding column of :
So
(b) By Theorem 3.4.9, is spanned by the basic solutions (Definition 3.4.8), one for each free column, read off from : and . As a check,
By (a), every column of is a combination of and , so
Exercise 3.6.8. Let and let be invertible. Decide which of the following are always the same for and for , with a proof or a counterexample.
- (a)The image .
- (b)The kernel .
- (c)The set of row vectors with (the relations among the rows).
- (d)The reduced row echelon form.
- (e)The rank.
- (f)Whether there is a left inverse.
Uses. Proposition 3.2.30 (Column operations preserve the image), Non-example 3.2.31 (Column operations change the kernel), Corollary 3.2.23 (Row equivalent matrices have the same kernel), Proposition 2.4.23 (Transpose of a product), Exercise 3.5.41, Definition 2.5.1 (Left inverse)
Techniques. Transpose to trade rows for columns (Technique 3.2.27)
Since and is invertible, a property that passes from to for every invertible also passes back from to . So in each “same” case it suffices to show one direction.
(a) Always the same, by Proposition 3.2.30.
(b) Not always. For and (which adds times column to column ), , and the kernels are and (Non-example 3.2.31).
(c) Always the same. Transposing, says , that is, . By Proposition 2.4.23, , and is invertible, with inverse . So by Corollary 3.2.23,
(d) Not always. is in RREF, and with , is also in RREF. They are different matrices, so their reduced forms differ (and so do their pivot columns).
(e) Always the same, by Exercise 3.5.41(c).
(f) Always the same. If , then
Exercise 3.6.9. Let and .
- (a)Show that if and only if for some .
- (b)Deduce that if and only if every row of is a linear combination of the rows of .
Uses. Theorem 3.3.10 (), Corollary 3.2.23 (Row equivalent matrices have the same kernel), Lemma 3.3.16 (Reading the columns of a matrix in RREF), Definition 2.3.8 (Matrix–vector product), Definition 2.4.1 (Matrix product)
Techniques. Reduce to RREF (Technique 3.3.22)
(a) Suppose . If , then . So .
Conversely, suppose . By Theorem 3.3.10, for an invertible , and (Corollary 3.2.23), so . Let be the pivot columns, write and for the columns of and , and let be the matrix whose column is for and for . We show , column by column.
Fix . By Lemma 3.3.16(a) (with ),
On the other hand, (Lemma 3.3.16(b)), so
satisfies . (If is itself a pivot column, then for and , so , and the argument still works.) Hence , that is,
(b) By the row description of a product (Definition 2.4.1, as in the second proof of Proposition 3.2.7), row of is times row of , plus , plus times row of . So “ for some ” says exactly that every row of is a linear combination of the rows of , and (b) is a restatement of (a).
If , then (b) applies in both directions, so and have the same row space; this generalizes one half of Exercise 3.3.50(b) to matrices with different numbers of rows. The proof of (a) is the second proof of Theorem 3.3.17 once more: it used only relations among the columns.
Uses. Exercise 3.2.42, Theorem 3.5.19 (Invertible matrices are products of elementary matrices), Theorem 3.3.10 (), Theorem 3.3.23 (Invertibility and RREF), Corollary 3.4.20 (Square systems), Proposition 3.3.8 (Properties of the output), Proposition 3.2.10 (Inverses of elementary matrices)
Techniques. Reduce to elementary matrices (Technique 5.4.6), Reduce to RREF (Technique 3.3.22)
Step 1: elementary matrices. By Exercise 3.2.42, for every , if has type 3,
Step 2: invertible . By Theorem 3.5.19, with each elementary. Applying (3.6.1) times,
With , and , this gives
Step 3: not invertible. Let with a product of elementary matrices (Theorem 3.3.10). Since is not invertible, (Theorem 3.3.23). A matrix in RREF with two pivots is (Corollary 3.4.20, (i) (iii)), so has at most one pivot and its second row is zero (Proposition 3.3.8(a)). Then the second row of is zero as well (row of is row of times ), so and , since a matrix with zero second row has . Now is a product of elementary matrices (Proposition 3.2.10), and , so Step 2 applied to gives
So
Step 3 is the part that is easy to forget, since a matrix that is not invertible is not a product of elementary matrices at all. In Section 5.4 the same strategy (elementary matrices, then invertible matrices, then a separate argument for the rest) proves
Exercise 3.6.11. Start with a real matrix. A move chooses a row or a column whose entries have a negative sum and multiplies every entry of it by .
- (a)Show that every sequence of moves is finite, whatever choices are made.
- (b)Deduce that some finite sequence of moves produces a matrix all of whose row sums and column sums are nonnegative.
- (c)Express the final matrix as , where is the starting matrix and , are products of elementary matrices of one type.
Uses. Proposition 3.2.7 (Row operations are left multiplications), Proposition 3.2.28 (Column operations are right multiplications)
Techniques. Algorithms as proofs: invariants and termination (Technique 3.3.5)
(a) Let be the starting matrix and the current one.
Invariant. Every entry of is the corresponding entry of . This holds at the start, and a move multiplies some entries by , which keeps it true. So is one of at most matrices (two choices of sign for each entry).
Monovariant. Let be the sum of all entries of . If a move negates a row or column whose entries sum to , the entries of that line now sum to , and no other entry changes, so changes by . So strictly increases with every move.
In a sequence of moves , the values
(b) Make moves as long as one is possible. By (a) this stops, and it stops exactly when no row and no column has a negative sum.
(c) Negating row is left multiplication by (Proposition 3.2.7), and negating column is right multiplication by the matrix (Proposition 3.2.28). Since
The proof follows the two halves of Technique 3.3.5. The monovariant alone would not do: an increasing real number can increase forever. The invariant supplies the finiteness.
- (a)Find all such that
for every . - (b)Deduce that
for every integer .
Uses. Definition 1.2.17 (Evaluation), Lemma 3.4.23 (Too many roots), Proposition 3.1.19 (Solution sets are translates of the kernel)
Techniques. Undetermined coefficients (Technique 3.1.9), Particular plus homogeneous (Technique 3.1.18)
(a) Write
so for every (Definition 1.2.17),
where
The condition is that
in the unknowns , in which does not occur. Solving from the bottom: , ,
and is the polynomial whose value at is , since
(b) Let be the solution with , so
The method needs a field in which comparing coefficients is allowed, which used four distinct points , and in which , , are nonzero, for the triangular solve. Over with both hold. Over and both fail: for instance vanishes at every point of , so a polynomial identity on the field no longer determines the coefficients.
Exercise 3.6.13. Let . Read indices modulo , so that and .
(a)Show that for every there is exactly one with
- (b)Now replace by . Show that the system
() has a solution if and only if . You may use Exercise 3.5.37.
Uses. Corollary 3.4.20 (Square systems), Exercise 3.5.37, Proposition 2.4.23 (Transpose of a product)
Techniques. Uniqueness gives existence (Technique 3.4.21), Extremal choice and minimal counterexamples (Technique 1.5.9)
(a) This is a system of equations in unknowns, so by Corollary 3.4.20 ((iv) (vi)) it suffices to show that the homogeneous system has only the trivial solution. Let satisfy
so and . As is the largest of the , every .
(b) Let be the coefficient matrix: on the diagonal, in the positions and (modulo ), and elsewhere. It is symmetric, , because exactly when . Let .
Necessity. Each occurs in three equations, with total coefficient (in equation with , and in equations and with ; these are three different equations since ). So summing all equations gives , and a solution can exist only if .
The homogeneous solutions are the constant vectors. Constant vectors are solutions, since . Conversely, let , and let be a largest entry. Then
Sufficiency. Suppose but has no solution. By Exercise 3.5.37 there is with and . Transposing (Proposition 2.4.23), , so for some , and then
So with the homogeneous system has only the trivial solution and every is reached; with there is a homogeneous solution , and in exchange one condition on . For a square system the two failures come together, as Corollary 3.4.20 says they must.
Exercise 3.6.14. For
Claim. For all there is exactly one with , , and .
Proof. The three conditions are linear equations in the three unknown coefficients of . The system is square, so by Technique 3.4.21 it has exactly one solution.
Uses. Corollary 3.4.20 (Square systems), Proposition 3.1.19 (Solution sets are translates of the kernel)
Techniques. Uniqueness gives existence (Technique 3.4.21)
The argument checks that the problem is linear and square, which is what makes Technique 3.4.21 applicable, and then skips the technique itself: the homogeneous problem must be shown to have only the trivial solution, and here it does not. For , the homogeneous conditions are , , and , and (that is, , , ) satisfies all three.
In fact, for every ,
So a solution can exist only if . If , then is a solution, and by Proposition 3.1.19 the solutions are