# Full text of "DTIC AD0616381: DYNAMIC PROGRAMMING AND CLASSICAL ANALYSIS"

## See other formats

AD616381 U wmuuhx corv wf i,^ 0 f 01 ^onf'-ON WHJ. bt ^ * "?S22^ RY osas or DDC. Rep'oduced by The RAND C o i p o r a t i o n • Santa Monica • C a I > ♦ o » n i a The views expressed ti this paper are not necessarily those of the Corporation CLEARINGHOUSE FOR FEDERAL SCIENTIFIC AND TECHNICAL INFORMATION. CFSTI INPUT SECTION 410.11 LIMITATIONS IN REPRODUCTION QUALITY OF TECHNICAL ABSTRACT BULLETIN OOCUMENTS. DEFENSE DOCUMENTATION CENTER (DOC) J I. AVAILABLE ONLY FOR REFERENCE USE AT DOC FIELD SERVICES. COPY IS NOT AVAILABLE FOR PUBLIC SALE. ^ 2. AVAILABLE COPY MILL NOT PERMIT FULLY LEGIBLE REPRODUCTION. REPRODUCTION WILL BE MADE IF REQUESTED BY USERS OF DDC. X' A. COPY IS AVAILABLE FOR PUBLIC SALE. B. COPY IS NOT AVAILABLE FOR PUBLIC SALE. 3 LIMITED NUMBER OF COPIES CONTAINING COLOR OTHER THAN BLACK AND WHITE ARE AVAILABLE UNT'jL STOCK IS EXHAUSTED. REPRODUCTIONS WILL BE MADE IN BLACK AND WHITE ONLY. TSL-I2I-2 65 DATE PROCESSED: V PROCESSOR: . / «■ * £ 7 ' SUMMARY P-1804 5-28-59 ii Over the Dost ten yearo, research In the field of dynamic programming has assumed many different forms. Sometimes, the emphasis has been upon Questions of formulation in analytic terms and concepts, sometimes upon the problems of existence and uniqueness of solutions of the functional equations derived from the underlying processes, occasionally upon the actual analytic structure of the solutions of these equations, some¬ times upon the computational aspects; and sometimes upon the applications—to control processes, to trajectories of various types, to operations research, tc mathematical economics. Inevitably, the result of this quasi-ergodlc behavior has been tc ignore a number of significant problems, and to treat a number of others In cavalier fashion. In this exposition, we wish to focus attention upon a number of interesting, difficult, and significant questions in analysis which arise naturally out of the functional equation technique of dynamic programming. Our aim la to show that this theory constitutes a natural extension of classical Investigations and that the corresponding problems are natural generallzatIona of problems of classical analysis. P-1804 0 - 28-59 1 DYNAMIC PROGRAMMING AND CLASSICAL ANALYSIS Richard Bellman 1. Introduction Over the last ten years, research in the field of dynamic programming has assumed many different forms. Sometimes, the emphasis has been upon questions of formulation in analytic terms and concepts, [1,2]; sometimes upon the problems of existence and uniqueness of solutions of the functional equations derived from the underlying processes, [2,5,4]; occasionally upon the actual analytic structure of the solu¬ tions of these equations, [5,6,7,8,9]; sometimes upon the computational aspects, [lO,ll]; and sometimes upon the appli¬ cations; to control processes, [2,12,15]; to trajectories of various types, [15,14] ; to operations research, [lp,16,17]; to mathematical economics, [18,19] • Inevitably, the result of this quasi—ergodlc behavior has been to Ignore a number of significant problems, and to treat a number of ethers In cavalier fashion. In this exposition, we wl3h to focus attention upon a number of Interesting, difficult, and significant questions in analysis which arise naturally out of the functional equation technique of dynamic programming. Our aim is to show that this theory constitutes a natural extension of classical, investigations and that the correspond¬ ing problems are natural generalizations of problems of classical analysis. As always in studying new areas, there is the hope that P-1804 light throw]', lr, VhU v-1'.v.ln territory will bo refiectad back upon still hidden parts .f t’».- s cIcihhVc-jI domain. We shall restrict our attention here to deterministic processes, reserving for other times any discussion of more complex and arcane processes arising fi-ora the study of stochastic and adaptive processes, [2]. These new areas of analysis where many different sub—disciplines such as algebra, topology, differential equations and probability theory merge and lose their separate identity in f 'll-embraclng problems offer bountiful and boundless regions for research. The reader familiar with the concepts and problems discussed here can readily construct for himself corresponding questions involving uncertainty. ?. Mnl ti.~t: Dec 1*4on PrccesHes Pyruir..r ; ro^i^nm ing is « mathematical theory of multistage decls 4 .r. prccer.sea. Txj uOttona «> r tM. theory may be P'und in [l] and [?J . Th t r v le:r m . w will he very much n :v neartir.yfui wl 4 ;: ir, the ccrt>»xt of dynamic programming. The rerder whc wishes, i owever, may \ -'jr . i e h 1‘ insertions of moti¬ vation and n’Jiir.l the p ir blem? *hr.t follow a? n ni’cdrw pulled cut ^ the bj;?. j. The Calculus of Variations Perhaps the most interesting and important example of a multistage lecislon process of continuous type is the calculus of variations. Consider the scalar variational problem of maximizing the integral ( 1 ) J(y) -,/ T Hx.yjdt, u 0 P-1804 9 - 28-59 3 over all functions y, where x and y are related by means of the dl f fpr**nM «^’j.itlon (2) St ' 0! ’'’ y) ' X ' 0) ’ =• Introducing the function (M f(c,T' - Max J'y), we obtain from the principle of optimality, [lj , the nonlinear partial dl ffer rt nt 1 ?.? oq’ntior. ( 4 ) jijr • Max P(c,v) ♦ G (c, v )ji- with the lnitla^ condition J f { C ,0 y * 0 . See [l ,2] . We ohall uso tula equation as a pivot for some of tno subsequent problems we ona*i preoent. 4. Existence and Uniqueness The equation in ,3.4) Is derived in a formal fashion, much as the Euler equation la customarily obtained. The first problem wo pose la that of finding conditions upon the functions P(x,y) and Q(x,y) which ensure the existence and uniqueness of a solution of tno unconventional nonlinear partial differential equation of (3.4). P-1804 9-28-89 4 One way to approach this problem is to employ the reeulte of the classical calculus of variations. These enable us to establish the existence of a solution of the original variation¬ al problem, under certain rather rigid conditions upon P(x,y) and 0(x,y). This is not a particularly satisfactory procedure for a number of reasons. In the first place, we would like to use Equation (3.4) to resolve the original variational problem. Secondly, we want to use analogues of (3.4) to study variational problems which partially or wholly escape the classical theory. This will be the substance of the following section. 3. Constraints A question of much analytic interest, and one which in any case is rudely thrust upon us by a great number of feedback control and trajectory processes, is that of solving the variational problem of (3.1) and (3-2) under the additional constraint (i) |y(t;| < k, o ^ t < t. It is certainly surprising that a simple condition of this nature should so effectively stymie the usual approach, but nonetheless, true. For a detailed discussion, see [2,20] . As a result of the presence of the foregoing restriction, the equation of (3.4) is replaced by the equation P-1804 £- 28-59 5 < 2 ' [ P(C ’ V) + 0 ( c ' v ) 39 - One might surmise that the presence of the constraint would simplify the investigation of existence and uniqueness of solution, as it certainly does the computational solution of equations of this nature. However, equations of the foregoing type have not been rigorously investigated as yet. 6. Discrete Version—I A standard route to the goal of existence of a solution of a functional equation Involving derivatives is uy way of difference equations. This approach is of the utmost importance in connection with the numerical solution of these equat ior.L . Let us then in place of (3*4) write the recurrence relation (l' ~ - f . m Max r?( c ,v) ♦ . f C c " 6 f T l 2b v L d0 where c and T assume respectively values of the form + kS, lb. Kb b —* 0, 6 —♦ 0, this relation formally approaches that of (3.4). The problem now is that of determining the connection be¬ tween the solution of (1) and the possible solution of (3.4), first under the assumption that (3.4) has a solution and secondly in the hope of using the solution of (1) to establish the existence of a solution of (3.4). Closely associated with questions of this nature is the P-1804 9 - 28 - 5 ^ 6 question of numerical stability of a computational algorithm such as (1). Finally, let us note that this is only one of a large set of possible discrete approximations to (3.4). 7. Discrete Version—II To obtain a discrete version of a different type, we use an idea of some importance as far as approximation techniques are concerned. In place of approximating to the exact equation describing the process, we can employ the exact equation for an approximating process. Hence, in place of the continuous decision process described by the equations of (4.1) and (4.2), let us consider the problem of determining the sequence jy^ which minimizes the function N-l (1) J K (y) * JJ'tvy-JA. k-0 where x,, and A are related by means of the equation (2) x k+l “ ' \ ’ a '-\’ y K' A ’ *0 = °> * " 0<1 ' 2 ’ ••• Here < X 1 X . y k - y (xfl) . n& - t. Setting (3) f N (c) - Min J N (y), y we obtain the recurrence relation ( 4 ) f„(c) - Max v F(c,v)& + f N_i( c ♦ . r-1304 9-28-59 7 with (5 ) f 0 (c) - Max F(c,v)A. J v Recurrence relations of this nature have been quite successful in the obtaining of computational solutions; see [10,14,21] . Observe that as A —+ 0, the equation in (4) formally reduces to that of (5*4). Yet, unlike (5.1), it involves only one small quantity A, and no ratio of small quantities such as A/6. T3iis la a most important point in connection with numerical stability. A small amount of work has be^n done on the subject of the rorve^hnce of the solution of (4) to the solution of (3.4); see [22,1,23#24] . Much remains to be done. 8. An Application We have mentioned above the possibility of applying ideas and techniques developed in the new field of multistage decision processes to classical equations. Let us give an example of this. Consider the partial differential equation (1 ) u t - uu x> u(x,0} - g(x), an equation which pussesces a shock discontinuity. In place of the usual finite difference scheme, a la (6.1), let us borrow the approach of (7.4^ and use the relation P-1804 >- 28-^9 (2) u(x,t ♦ A) - u(u ♦ u(x,t'iA,t), as an approximation to (1). Excellent results have been obtained, even In the Immediate neighborhood of the shock, using this algorithm; see [253* No work has been done, however, on the question of tne convergence of the solution of (2) to the solution of (1) as & —* 0, for this equation or for more general equations. For another application of this Idea, see [26] . 9. Approximation In Policy Space Another attack on the basic problem of establlsning existence and uniqueness of solutions of the equation In (4.7, or (o.2) is by means of the method of successive approximations. In place of the usual approach, let us Invoke the technique of approximation In policy space ; see [\ , 2 ] . We begin by guessing an Initial policy , v^ » v^(c,T), and using this policy to determine a return function fg • fQ(c,T), by means of the linear partial differential equation (1) yr " F(c *V * 0 (c ' v o , ?c fi ' f o (c ' 0; " °* Having determined fQ(c,T', let us determine the function v^ by the condition that It maximize the function P-1604 9-26-59 9 Using this new policy—function, let us determine the new return function f 1 by way of the linear partial differential equation b T (3) ■ P(c,v^) ♦ Ofc.v^^l, f 1 (c,0'i • 0. Continuing in this fashion, we obtain a sequence of functions ^ f^ | and iv^J. I n view of the manner in which is deter¬ mined, it is clear that 3f 0 W F(c,v Q ) af o * Q ( c » v 0^^c~ 2f, < F(c,v 1 j + Prom this, we would expect to find that f Q (c,T) < f 1 (c,T) for T > 0, and this is actually the case. Oenerally, the merit of this approximation method is that it yields monotonic behavior, (3) f 0 (c,T) < f 1 (c,T ; < ••• < f n (c,T) < •••. important problem is to determine conditions under which this sequence converges to a solution of (.5.4}, and, of course, to investigate th* application of this technique to more general equations. Some preliminary results for a much simpler problem may be found in [l] , Chapter 11. 10. Positive Operators A crucial result in the foregoing procedure is the con¬ clusion f 1 > fg for T > 0 as a consequence of the equality P-1804 9-28-59 10 and the inequality 3f (2 J » v i ^ ♦ G(c » v l j Jc^' granted common initial conditions at T « 0. Questions of this nature are part of the theory of positive operators. Oiven a relation Au 2 0» where A is an operator, we wish to determine when tnts implies u > 0. Thi) type of investigation was initiate! by Caplygin, [27]; see [28,29,30j for further results and references. Again much remains to be done In this field. 11. Quaslllnearlzation Let us now indie it* another aprl ‘ oat i< >n of new technlpj'*?' developed in the theory of dynamic programming to classical analysis. We have Just seen, in ^9 , that equations of the special form of (3.4 1 can be approached along a route which is not open to the equations of classical analysis. In view of the raonotonlclty of approx.LT.ctlo 0 , 1 '»«?ry '»aiu:-*bie analytic and computational aid, it may be worth devoting some effort t the question of converting an equation of conventional type into one of the form of (3.4). What wti are doln* Is transforming an equation arising from a descriptive process into one which arises from a variational process Tills, of course, is a familiar Idea in analysis and one of great power and versatility. The way in which we do it is, however, new. To give a simple example of the technique which can be P-1804 9 - 28-59 11 used, consider the Rlccatl equation (1) u* - u 2 + a(t), u (0) - c . Write (2) u 2 - Max (2uv - v 2 ), v so that (1) assumes the form (3) u' - Max |^2uv - v 2 ♦ a(t)J , u(0) ■ e. Consider the related linear equation (4 ; IT' - 2Uv - v 2 + a(t), U(C) - c, whose solution we write In the form (5) U - T(v.t). We suspect that (5) u - Max U • Max T(v,t), v v a fact which la a consequence of the positivity of the operator d/dt - 2v, In the sense of §10. Since we can obtain an explicit representation for T(v # t) In terms of Integral equations, (o) furnishes an Interesting representation for the solution of (1). Oenerally, If f(u,t) Is convex as a function of u for 0 < t < tQ# we can write (7) P-1304 9-28-59 1? f(u,t) - Max [f(v f ti ♦ ( u - vjf y (v f t)J, and if f (u,t) is concave, we can write (8) f(u,t) - Min ff(v,t) 4 (u - v)f (v,t) . v 'Riese representations enable us to treat differential equations of the form (9) u' - f*u,t ), and, more generally, to transform functional equations of the form (10) Au - f (u,t ), where A la an operator, into quasllinear equations of the forra (11; Au - Max v Having obtained this form, we can employ approximation in policy space, and proceed as above. This representation also has important computational advantages. For a more detailed discussion, with many examples, see [j?B, 29 j- 12. Semi—Qroup Theory The modem theory of semi—groups of transformations, [^l}, deals with functional equations of the form : f ( V , t ) 4 ( U - V i f ( V , t )j T P-lbO* 9 - 23^9 19 (1) 3T " Au ' where A Is a linear operator. A particular equation of thla type is the linear partial differential equation of (9.1). The functional equations associated with dynamic programming pro— ceases of continuous type have the form (2) - Max [A(v)u + b(v)] . In view of tne fact that (2) contains (1), we can expect a diffusion of knowledge in both directions. In the first place, in view of the quaalllnearlty of (2), we may expect that a large part of sem.*,.-group theory can be applied to the question of the existence of solutions of (2). In this way we would hope to obtain far stronger results in the calculus of variations than those currently available. In particular, we would aspire to a more complete theory for variational problems with constraints. Secondly, we may expect to utilize the results of the linear theory, and some of the results and techniques of dynamic programming, tc obtain a theory of nonlinear equations whicn can be written in the font, of (2). 13. Multidimensional Variational Problems Consider tne problem of maximizing the Integral- ( 1 ) J(u^ •Jpt (u,u x ,u y )dA, where the integration is over the interior of a two-dimensional P-1804 9 - 28-39 14 region R, and the value of u Is prescribed on the boundary B of R, (2) u - g(P), P * B. In a procedure completely analogous to the one—dimensional case, we may introduce the functional (3) f(g(P);R) - Max J(u). u Considering a sequence of shrinking regions, it Is not difficult to obtain a functional equation for f(g(P);R). The derivatives that occur will now be functional derivatives, cf. Although this technique ha3 been used to obtain the Hadamard variational formula for the Green’s function of a region, and other results, [35,54,53,36], no work has been done on the existence and uniqueness of solutions of equations of this nature. An Interesting side problem associated with variational questions of this nature Is that of determining the functional form of f(g(P);R). Sven for the classical problem involving the Dirlchlet functional, (») J(u) 4 u>A, deep and subtle analysis Is required; see Osborn, ([57], where further references may be found. P-i604 9-28—59 15 14 . Stability and Asymptotic Behavior Aa soon as the fundamental questions of existence and uniqueness of solution have been disposed of, we can turn to the deeper and more Interesting problems of the analytic structure of the solution. In particular, we would like to examine the stability properties of the solution and its asymptotic behavior as t -» oo . A small amount of work has been done in this direction, but no systematic theory has been constructed; see 'i7,38,59] . A theory of this nature can be based upon an extension of the present theory of positive operators; see [40,41,42]. 15. Implicit Variational Problems Sc far we have considered variational processes of fixed duration. It is of interest to consider processes in which the upper limit T depends upon the policy used, and those of still more general nature . Those are examples of implicit variational problems. One of the most important examples of this type of problem la furnished by "bang—bang" control. Suppose that we have a system S ruled by a vector differential equation (1) - g(*,y), x(0) - c, where the components of y are subject to the restrictions \& / l y ± V -) 1 js ^ ^ ^ " 1 » 2, ... ,N, or P—1804 9-2S-69 lo (b) y 1 (t) - + k 1 , 1 - We wish to choose y In such a way as to minimize the time required to force tne system from the initial state c to nother state, say 0. Por the linear case, (3) • Ax -f y, x(Q) - c, a threat deal has b r r*n a^ne U31 n _j k variety >f technl :ju'*3; see [43,^4,4for further' references. 10. Purt’ne r Directions It is easy to pose a number of additional questions of interest if we admit two person processes, multistage games; cf. [4,46], and If we Introduce stochastic and adaptive pro¬ cesses in general; [1,2,39*47] . The range of investigation Is now so broad, and so different in many ways from the classical tableau, that we feel It better to present 113 cuss ions of this nature in separate publications. P-1 ft 04 9—26-69 17 RSRI-lRtfNC 1. P. VI Ir.fij , JynamV F'ix r i’anmlng , Princeton Univ . Press, Princeton, ri7 77 , 19**. 2 . -, Dyrm.. 1 r jgrammlng and Adaptive Control Pro— cesses , to appear. 3. -, ”50106 functional equations In the theory of dynamic programming—I: functions of points and point transformation," Trans, Amer. Math. Soc .. vol. 80, 1955# PP. 51—71. 4 . -, "Functional equations in the theory of dynamic programming—III," Rend. Clr. Mate. Palermo , ser. 2, tomo 5# 1958, pp. 1-23. 6. -, "A variational problem with constraints In dynamic programming," J. Soc. Ind, Appl. Math ., vol. 4, 1950, pp. 48—ol. ' ’ ' " o. -, "On a class of variational problems," Q. Appl. Math ., vol. 14, 1957, pp. 353-359. 7 . -, "Functional equations in the theory of dynamic programming—V: positivity and quas 1-1 inearity," Proc . Hat. Acad. 3c1. USA , vol. 4l, 1955, PP. 74>-748. 8 . R. Bellman, I. Ollcksberg, and 0. Gross, "On the optimal Inventory equation," Management Scl ., vol. 2, 1955, pp. 8>-104 . 9 . R. Bellman, "On some applications of dynamic programming to matrix theory," Illinois J. Math ., vol. 1, 1957, PP. 297-501. 10. R. Bellman and S. Dreyfus, Computational Aspects of Dynamic Programming, Princeton Univ. Press, Princeton, k. j to appear. 11. -, "A bottleneck situation Involving Interdependent Industries," Naval Research Log. Q .. vol. 5 , 1958 , pp. 21 - 28 . 12. R. Bellman, S. Dreyfus, and R. Kalaba, "Dynamic programming trajectories and space travel," Proc. Fourth Symp. on Astronautics , Los Angeles, 1959# to appear. 15- R. Bellman and S. Dreyfus, "An application of dynamic pro¬ gramming to the determination of optimal satellite trajectories," J. British Interplanetary Soc ., to appear. P-1804 9 - 28-69 18 14. T. Cartalno and S. Dreyfus, "Application of dynamic pro¬ gramming to the airplane minimum time—to—climb problem," Aero, Sng. Re^ ., 19L>7 • 15. S. Dreyfus, "a note on an Industrial replacement process," Operational Research Q ., December, 1957. 10. R. Bellman, "Equipment replacement policy," J. Sue. Ind . Appl. Math ., vol. 3, 1955, pp. 153-136. 17. R. Howard, Discrete Dynamic Programming Processes , Thesis, Massachusetts Institute of Technology, -. 18. R. Beckwith, Analytic and Computational Aspects of Dynamic Programming Processes of Ttlgh Dimension , Ph.D. TheBla. Purdue University] 1 ) 59 . 19. R. Bellman, I. Qlicksberg, and 0. Gross, "On the optimal Inventory equation," Management Science, vol. 2. 1955. pp. 83-104. 20. R. Bellman, "Dynamic programming and its application to tt.e variational problems In mathematical economics," Proc . S ymp. In Calc il *s of Variations and A^licatlona, Arner. Math. Soc7, Chicago, April, i"$£b, pp. II 5 —published by McGraw-Hill Book Co., Inc., New York. 21. R. Bellman and S. Dreyfus, "On the computational solution of dynamic programming processes—I: on a tactical air warfare model of Mengel," J. Oper. Re3 ., vol. o, 1956, pp. o5-7 1 . 22. R. Bellman, "Functional equations in the theory of dynamic programming—VI: a direct convergence proof,' Ann. Math ., vol. 03 , 1957, pp . 215-225. 23* H. Osoorn, "The problem of continuu .u programs," Pacif ic J. Math ., vol. o, 1950, pp. 721-/ , 51. 24. W. Fleming, Discrete Approximations toSeme Continuous Dynamic Programming Process es, The RAND Corporation, Research Memurand in RM-15C1, June 2, 195:'. 25. R. Bellman, I. Cherry, an! 0. M. Wing, "A note on the nomerlcal Integration of a class of nonlinear hyperbolic equations," Q. App1. Math ., vol. lo, 1958, pp. 151—163. 2o. R. Bellman, "On the nonnegativity of solutions of the heat equation," B 11. Unlone Mateiaatlco , vol. 1?, *h'' T , ; : . 529- 25- P-1804 9 - 28-59 19 27. S. Caplygln, A Hew Method i‘or the Approximate Integration of Solution of Differential Equations , Princeton unlv. ?ress, Prlnoeton, ft. J., to appear; translated by R. Bellman and R. Kalaba. 28. R. Bellman, "Functional equation* In the theory of dynamic programming—V: positivity and quail-linearity," Proc. Nat. Acad. Sci. USA , vol. 41, 1955, pp. 743-746. 29. R. Kalaba, "On nonlinear differential equations, the maxi¬ mum operation and monotone convergence," J. Nath, and Mech., vol. 6, 1959, PP- 519-574. 30. E. F. Beckenbach and R. Bellman, Inequalities , vol. 1, Krgebnlsse der Math., to appear. 31. B. Hllle and R. Phillips, Functional analysis and semi¬ group s , Amer. Math. Soc. tiolloq. Publ., vol. 3i, 1958. 32. R. Bellman, "Dynamic programming and a new formalism in the theory of integral equations, Proc. Nat. Acad. Sci. USA , vol. 41, 1955, PP- 31-34. 33* R- Bellman and H. Osborn, "Dynamic programming and the variation of Oreen's functions," J. Math, and Mech., vol. 7, 1958, pp. 81-86. 34. R. Bellman, "Functional equations in the theory of dynamic programming—VIII: the variation of Oreen's functions for the one-dimensional case," Proc. Nat. Acad. Sci. USA, vol. 43, 1957, pp. 839-841. ~— 35* P • Bellman and S. Lehman, "Functional equations in the theory of dynamic programming—IX: variational analysis, analytic continuation, and imbedding of operators," Proc. Nat. Acad. Sci. USA , vol. 44, 1958, pp. 905-907. 36. -, "Functional equations in the theory of dynamic programming—X: resolvents, characteris¬ tic functions and values," Duke Math. J .. to appear. 37. H. Osborn, "The Dlrlchlet functional," J. Math. Analysis and Appl ., to appear. 38. R. Bellman, "A Markovian decision process," J. Math, and Mech ., vol. 6, 1957, pp. 679-t>84. ~ —* 39. -, "Directions of Mathematical research in non¬ linear circuit theory," PQCT Trans ., to appear. 40. M. Kreln and M. A. Rutman, Linear Operators Leaving Invariant a Cone in Banach Space , Amer. Math. Soc. Translation No. 26, 1950. P-1804 9-28-89 20 41. T. S. Harris, Branching Processes , Jtrgebnlsse der Math., to appsar. 42. R. Bellman, Introduction to Matrix Analysis , McGraw-Hill Book Co ., Inc., New Vork. 19^9* 4). R. Bellman, 1. Glickaberg, and 0. Oross, "On the 'bang-bang' control problem," Q. Appl. Math ., vol. 14, 19bb, pp. 11-18. 44. J. P. LaSalle, "On time optimal control systems," Proc. Nat. Acad. Scl. USA , vol. 4?j, 1989, pp. 87>677. 4^. R. Bellman and J. M. Richardson, "On the application of dynamic programming to a class of implicit variational problems," Q. Appl. Math ., to appear. 4b. Studies in Qame Theory—-!, II, III , Annals of Math. Studies, Princeton University Press, Princeton, N. J. 47. R. Bellman and R. Kalaba, "A Mathematical theory of adaptive control processes," Proc. Nat. Acad. Scl. USA, to appear.