UNCLASSIFIED r lieoAoiutced. hu the ARMED SERVICES TECHNICAL INFORMATION AGENCY ARLINGTON HALL STATION ARLINGTON 12, VIRGINIA UNCLASSIFIED NOTICE: When government or other drawings, speci¬ fications or other data are used for any purpose other than in connection with a definitely related government procurement operation, the U. S. Government thereby incurs no responsibility, nor any obligation whatsoever; and the fact that the Govern¬ ment may have formulated, furnished, or in any way supplied the said drawings, specifications, or other data is not to be regarded by implication or other¬ wise as in any manner licensing the holder or any other person or corporation, or conveying any ri^ts or permission to manufacture, use or sell any patented invention that may in any way be related thereto. ME MORANDUM RM-3245-PR AUGUST 1962 (Zil'J cs. l-v. ■ c:) C'- cri «cX, '■ I- c,:’- ''-X FACTORS IN SELECTING AND TRAINING PROGRAMMERS Anders Sweetland PREPARED FOR: A Q T I / I V ■ f UNITED STATES AIR FORCE PROJECT RAND 7^ RiinD SANTA MONICA • CALIFORNIA- 283 796 MEMORANDUM RM-3245-PR AUGUST 1962 FACTORS IN SELECTING AND TRAINING PROGRAMMERS Anders Sweetland Th is research is s[)oiisored hy the I'nited States Air Force under I’roject HAND —Con¬ tract No. AF 49(6dd)-T()0 inonitorcd l>y the Directorate of Development Planning, Deputy Chief of Stall. Heseari h and Technology, Hq I'SAF. Views or conclusions con¬ tained in this Memorandum should not he interpreted as representing the olTicial opinion or policy of the United States Air Force. Permission to quote from or reproduce portions of this Memorandum must be obtained from The RAND Corporation. -iii- PREFACE A request from the Coii5)troller' s Office, SEAM, for assistance with the selection and training of programmers used in Electronic Data Processing work, provoked a reconsideration of a number of findings the author had uncovered in the programmer training program at the System Development Coiporation. The findings should he of interest to those groups or organizations who enploy programmers ,•>< or are considering the development of an electronic computer instal¬ lation. Although the results presented in the Memorandum are preliminary, they do point both to the paucity of conclusive research on this inportant occupation and to the desirability of further studies aimed at the selection and training of programmers for all types of EDPE installations. *It is necessary to understand that this study is concerned with programmers. Programmers are people who control the behavior of Electronic Data Processing Equipment (EDPE). Their activities are quite different from those people who control the behavior ol Electric Accounting Machinery (EAM). (Also called Punched Card Accounting tiachinery ... PCAM.) -V- SUMMARY This study atten^jts to determine some of the factors related to the selection and training of computer programmers. Section I describes the evaluation of nine classes of programmer trainees according to their intelligence, motivation, and classroom performance. Supervisors' ratings were also obtained as a follow-up study. The findings show that both intelligence and motivation, par¬ ticularly motivation, are good predictors of classroom performance. Intelligence is also a predictor of supervisors’ ratings, but not as good a one as classroom showing. (The data did not permit testing the relationship between motivation and supervisors’ ratings.) The results described in Sec. I suggested that it would be profitable to explore non-cognitive measures. Section II deals with a study of one such measure: the vocational interest inventory. The investigation showed that programmers have interests that clearly distinguish them from the lay population. As a result, a scoring key for the Kuder Vocational Preference Record was developed. It is desirable to en^jhaslze that this key should not be considered a final product, however, but rather as an illustration that such a key is feasible. Section III discusses the potential fruitfulness of research in programmer characteristics, interests, and aptitudes; it sxiggests four areas for such research: (l) the organismic factors, with emphasis on characteristics other than intelligence, ( 2 ) programmer svipervi- sors, (3) training, and (h) the working milieu. CONTENTS PREFACE . lil SUMMARY . V Section I. RELATIONSHIPS AMONG TRAINING, INTELLIGENCE, AND MOTIVATION . 1 II. AN INVESTIGATION OF A NON-COGNITIVE MEASURE . 8 III. SUGGESTIONS FOR FURTHER RESEARCH . I 5 IV. CONCLUSIONS . I 8 - 1 - I. RELATIONSHIPS AMONG TRAINEES. INTELLIGENCE AND MOTIVATION About 195^> the Numerical Analysis Department of the RAND Corporation felt the need for more formal selection procedures to meet their burgeoning demand for computer programmers. They approached the System Research Laboratory now the System Development Corporation (SDC) for assistance. As a first step, members of the department underwent a battery of tests and were also ranked (subjectively) for programming ability. The tests were correlated with the rankings, and a multiple regres¬ sion analysis made. (The multiple correlation coefficient was 0.59)- The analysis yielded fo\ir tests that accounted for most of the explainable variance. Thurstone Primary Mental Abilities (PMA) Test: 1. Verbal 2. Reasoning 3. Spatial Thurstone Temperament Schedule: 4 E (emotional stability). Both RAND and SDC have used this set to date, with minor variations. Both have dropped the E scale (which is awkward to explain and to administer). Both override the percentiles cut¬ offs if the applicant has compensating qualities (such as extensive background in mathematics, or programming experience). SDC requires that applicants have had at least one course in calculus. Since the major components of most intelligence tests are measurements in the verbal-reasoning axea, it is a I'easonable - 2 - conclusion that the two companies are limiting their hires to approximtely the upper 2 to 5 per cent of the population in general intelligence (roughly, those with IQ's of 120 and up). Although the use of the PMA is a little hard on t he recruiters, the system works well: fewer than 5 per cent of terminations are due to lack of ability. As will be shown, however, intelligence is not the sole determinant of programming success. Shortly after setting up the training program at SDC, we dis¬ covered that the PMA scores could not be used for counseling trainees; there was insufficient spread (as a result of the two 90 th-percentile cut-offs used in hiring). It was impossible to determine the relative level of Intelligence of the individual student. We desired a test that would: (1) Measure in the same dimensions as the PMA; (2) Distinguish among trainees; and (3) Be easy to administer and score. The Otis-ttlgher Examination, Form D, met all three requirements. A factor analysis (coupled with other tests) showed that it measured the same dimensions. A reduction in testing time to 20 minutes yielded a spread from 25 to 75 in the raw scores. It could be administered to a group in less than 25 minutes, and scoring took less than a minute for each person tested (see Fig. l). The measure of intelligence (unless stated otherwise) in the remainder of this study is the raw score obtained on the Otis Form D taken under a 20 -minute time limit. The programming classes were composed of approximately 20 students each. Each class ran eight hours a day and lasted eight weeks. As Fig- 1 — Scattergram of Otis Scores (20-Minute Time Limit) each unit of work was completed (lasting 3 to 5 days), the students were tested for knowledge in that unit. A constant program of item analysis was carried out. After the first four or five classes, the test items stabilized at the 50 -per-cent level; i.e., each item was answered correctly by half the class. In the comparisons that follow, the measures of performance are based on the total scores for the eight-week training period. These vill be either the raw scores, or the raw scores converted to rank-orders. At the time, we were working on some characteristics of human motivation, and decided to include part of this work in the present study. Two estimates of motivation were used (with motivation defined here as a desire to lefitrn programming): (1) Estimates from instructors . Each class had two instructors. At the end of the course, each ranked the students independent!;,''. The lists were then compared and discrepancies resolved by discussion, which resulted in a third ranking. This third ranking was one of the measures used in the comparisons• (2) Estimates by peers . Each student ranked the other nxjmbers of the class (omitting himself). The rankings were first tested for stability, (using Kendall's Coefficient of Concordance*) and the composite rankings were rank-ordered. This composite ranking is the measure used in the comparisons. Because the peer estimates proved to be much more stable than the instructor estimates, we * M. G. Kendall, Rank Correlation Methods , Chsirles Griffen & Co., Ltd., London, 1955- -5- Table 1 CX)RREIATION BETWEEN UnELTIGENCE AND CLASS PEePORMAECE AMD MOTIVATION AND CLASS PERFORMANCE Class Intelligence: Otis & Grades Motivation Inst & Grades Peer & Grades 3 0.31 0.34 4 0.30 0.66 5 0.59 0.82 6 0.37 0.49 0.71 7 0.23 0.46 0.82 8 0.44 0.74 9 0.30 0.67 10 0.44 « 0.57 11 0.60 0.61 Mean 0.40 (?»$9 NOTE: Instructor and peer rank¬ ings were used ais estimation of mo¬ tivation. Kendall's Tau was used as the correlation meastire. (Since the experiment began with Class 3^ 1 and 2 are not included.) A Tau of 0.27 is significant at the 0.05 level for N = 20, the average class size, finally used them exclusively. The relationship among intelligence, motivation, and class grades is shown in Table 1. In this compeirison, the Otis scores and class grades were converted to rank orders. Kenall's Tau was used as the statistical test. (To those more accustomed to the conventional Pearson product-moment, a Tau of O. 3 O is about the same as a Pearson product- moment of 0 . 45 ). Since the time these data were collected, a number of studies relating intelligence and performance have been compiled. These are summarized in aui excellent paper by Dallis K. Perry, Computer Progrft»nm<»T- Selection Testing , Field Note 6371, System Development Corporation, Santa Monica, California, MEirch, 1962. ** Kendall, 0 £. clt . - 6 - The interesting peurt of these data is the ability of motivation scores to predict grades. In eveiy instance, the estimations of motivation predict grades better than do the fonnal measures of intelligence. It thus appears that it would be profitable to investi¬ gate characteristics other than intel l igence, a conclusion bolstered by the following observations. At one time, as part of a salary review, each supervisor ranked his staff according to their "value to the company" and, this done, was asked to convert his ranking to a company-wide percentile ("Of all the programmers in the conpany, where would you place this person, percenteige-wise?" ). As a part of the quality improvement of the training program, we collected these supervisors* ratings. Admittedly, they are imprecise. They came from several points in time: sometimes shortly after the programmer was on the Job, sometimes after he had been there several months. The supervisory experience was equally varied, nor was it possible to get 100-per-cent samples. Thus the supervisors' ratings lack the nicety of laboratory data, but at least they are real-world measure s. We matched the ratings with the Otis scores and training scores and correlated the samples (using Pearson product-moment). Since each sample (N = 50) is a mixture of several training classes, no direct conparlson can be made with the previous data. See Table 2. Two conclusions appear warranted: (l) A student's showing during his training is important to an organization hiring programmers; it predicts his ratings by immediate -7- supervlsors. This fact in turn implies that the training program per se is (or should be) a major concern of the organization. (2) Programmer selection might be improved by investigating areas other them intelligence. The Otis test does a reasonably good Job of predicting class¬ room showing; and the classroom is a surprisingly good predictor of the supervisor's rating. 'Ibe Otis test does not predict the super¬ visors' ratings very well, however. Table 2 CORREIATIONS AMONG OTIS SCORES, TRAINING SCORES, AND SUPERVISORS' RATINGS Otls- Grades Grades- Ratings Otis Ratings 0.46 0.66 0.26 0.40 0.42 0.15 0.25 0.48 0.01 0.49 0.73 0.29 0.47 0.32 0.27 0.27 0.32 - 0.11 o. 4 o 0.54 0.29 0 . 4 l 0.28 0.09 Mean 0.39 0.’'-7 0.16 NOTE: A correlation of 0.27 is nificant at the 0.05 level for N = 50. - 8 - II. AM INVESTIGATION OF A NON-COGNITIVE MEASURE Section I suggested that exploration of the-non-cognitive areas might improve the selection of potential programmers. During the interview, the recruiter faces one central question: whether to hire or not; to answer it, he needs objective measures that have predicted success in programming. Intuition does not help. F„r example, one would intuitively surmise that an applicant's mathematical background should foretell his success in programming. Experience proves other¬ wise. A sample of 40 cases was used as a test, in which semester hours of mathematics (which ranged frcm 6 to to) were correlated with class grades. The correlation was -O.I 7 (not significant). Vocational backgromd was equally unfruitful. So was the amoimt of college that applicants had had: some of the poorest students were Ph.D's, but so were some of the best. As in most professions, there were few outstanding females, but females (again as in most professions) were similarly rare at the low end of the continuum. Other than the intelligence test, our fictive recruiter has little to help him at present. These negative considerations lead to the following rationale. Most vocational guidance programs encompass three areas of measure¬ ment, two of which have been considered: (l) ability, (2) achievement, ( 3 ) interest. The intelligence test measures ability and does nredict success in programming. The simple and obvious achievement measxu:es (amoimt of college, amount of mathematics, etc.) yielded - 9 - little. Finally, the area of interest-measurement has not been explored at all. It is an appealing one for two reasons: (1) A vocational interest score is believed to be a low-key measure of motivation; and (2) The data of Sec. I indicate that motivation plays a major role in success in programming. The following is a description of an exploratory study in vocational interest testing. The Kuder occupational preference record was given to 100 programmer trainees. Since trainees are not professional programmers, we had misgivings about using these data to establish a preference profile; however, it is comparatively easy to get data from trainees and very difficult to get data from pro¬ fessionals. Consequently, our first question was: Will trainees reasonably resemble professionals in their responses to the KvuierV The hundred trainees were split randomly into two groups. A Kuder key was established by standard procedures. (Essentially the procedure was to ask whether the proportion of the experimental group who answered an item in a specific manner differed significantly from the nomnative group proportion. If so, the item was incorporated in the scale. ) In this st\idy, an item had to distinguish (from the normative group) at the 5“Per-cent level in both experimental groups to be Included. This procedure established a scoring key for trainees. The next question was: Do professional programmers answer the K\ider in There is one unexplored area, however: success in medical school has been predicted by combining college grades in a specific set of related subjects. It may be that combinations of grades would have similar predictive value for programming success. - 10 - the SEune way trainees do? The answer sheets of the 100 students and of 30 BAND programmers were scored with the key constructed from the trainees' answers. In addition, 100 answer sheets from the general population were obtained and were scored with the same key after being split into two random groups. The results of this manipulation are shown in Figure 2. The reader can see from Fig. 2 that trainees and professionals answer the Kuder in essentially the same manner, and that their interests (as measured by the Kuder) are quite different from those of the general population. Following this happy discovery, we made several trial-and-error attempts to find a scoring key that would yield the least overlap among the three samples of programmers and the two samples from the general population. The resisting key was obtained by isolating those items that distinguished programmers from the lay population in all three of the experimental groups (i.e., to be included, an item had to distinguish in each of the three separate samples). The emswer sheets were rescored with this key. It is obvious that pro- grEunmers have interests different from those of the lay population. (See Fig. 3 for resvilts.) Two precautions are in order: first, although the area of vocational preference appears to be promising, there has been no validation for this key; and although programmers can be distin¬ guished from the general population in terms of their interest, it remains to be demonstrated whether these interests are related to success in programming. (For example, if we partial out intelligence. - 11 - Fig. 2 — Kuder Scores, First Trial: Scattergrams of Three Programmer and Two General Population Samples - 12 - 100 SDC + 30 RAND 100 Norm 50 49 48 47 46 45 // 44 1 43 42 THt 41 HU 40 III 39 mt IHJ ifi 4 38 mm If 37 mm mm 36 numii 35 m m Iff / 34 rHl HU // 33 HU tru rm 32 rm m m 31 HU HH 30 H 7m TtM 1 29 HH HH / 28 III ITH / 27 HH 26 / HH HH 25 fUt H 24 mi H 23 HH HH 22 HH 21 rm HH 20 HU ! 19 / 18 H 17 H/ 16 / 15 M - 35.684 M - 25.410 O'- 3.646 O'- 4.338 Fig. 3 — Final Kuder Key Scoring, Comparing 130 Programmers with 100 General Population -13- will the Interest scores be related to success In training, success on the Job, or continxiatlon in the profession of programming?) Until the key has been validated, it is desirable to follow the more conservative approach used in the VA program (Public Law #l6 and Public Law following World War II. In the VA program, unless a person showed a career interest above the 751 h percentile of the general populatlcxi, it was deemed that his interests were not crystallized enough to warrant his planning for that career and he was encouraged to explore further. (We may note in passing that a 75-per-cent cut-off of the general population would have cut off the lower 7 per cent of the programmers used in this study.) Second, unlike intelligence (which is fixed at conception), interests are malleable. It is generally true that a person's vocational interests tend to stabilize when he is about twenty-five, but this is largely a cviltural happenstance. A person can become interested in a new profession at the age of 50. This is smother way of seiying that vocational guidance, based on interest testing, should operate only in the crudest of dimensions: a man with an interest in science and mathematics and none in social services should presumably be steered toward the engineering and related professions, where he is more likely to find a stimuJ-ating occupa¬ tion, and away from the social-service professions. Because interests are learned, however, it would be incorrect to assume that such a man cannot become interested in social services. To summarize this Section: Programmers ajxpear to have a set of homogeneous interests that distinguish them from the general population, and the selection of programmers mi^t be improved by measuring these interests. Interest scores should be used conservatively, however, until interest is shown to be a valid predictor of programming success. If interest measures are to be used, it is suggested that the upper 25 per cent of the general population be used as a recruiting cut-off point. -15- III. SUGGESTIONS FOR FURTHER RESEARCH The explosive growth of computer technology in the past ten years indicates that the profession of programming warrants consider¬ able exploration. Computer installations are expensive, and what we get out of them is highly related to the talents of the programming staff. It is worthwhile to begin exploring the dimensions of pro¬ gramming talent. The programming activities of most large computer installations fall into two broad categories: (1) Service functions . These are the bread-and-butter jobs: payroll, billing, insurance premium computations. Essentially, ex¬ ternal needs define the programmer's actions. The Jobs generally make considerable demands on his ingenuity, but rarely necessitate inspired, creative imagination. (2) Research in programming . In contrast to the bread-and- butter tasks, basic research is another world. In it, people try to get the computer to do things Turing never had in mind, such as translations of language, learning, concept formation, problem¬ solving (in the psychological sense), and abstract writing. Here imagination is a sine qua non . Intuitively, we would surmise that these two types of activities demand two different types of programmers, although we do not know what their distinguishing qualities may be. (We may be wrong, however: perhaps the same programmer can double in both types of activities.) In one instance, the program is a means to an end (as statistics are -16- used in psychological experiments); in the other the program is an end in itself (e.g., the development of linear programming). Research on this question is advisable. A second area for exploration is determination of the qualities of good programmer supervisors. From a company's point of view, good supervision is critical; but how does the recruiter respond to the request, "Of the ten programmer trainees you recruit, include two who are potential supervisors?" More basically, what qualities (if any) distinguish good programmers from good supervisors? A third area for research is training. One question is how old (or young) a person should be when he begins to learn programming. Some of the work on concept formation suggested that the ability to form concepts is independent of chronological age but directly related to mental age. The same may be true of programming. Extrapolating the Otis results suggests that the brighter (upper 5 per cent) juniors and seniors in high school could be trained in programming and benefit by the process, but that it would be an unusual high school freshman who could do as well. Psychologists recognize this as a variation of the maturation-versus-learning problem. Finally, there is the comparatively unexplored area of the work¬ ing milieu. If it is true, as our data imply, that the programmer is a different sub-species, perhaps he needs a special kind of environ- * ment in order to be productive. The work of Pelz on the productive *1). Pelz, "Some Social Factors Related to Performance in a Research Organization", Adm. Scl. Quart ., Vol. 1, No. 3, December, 1956 , pp. 310 - 325 . -17- milieu suggests that the scientifically productive working environment differs dramatically from what one might guess (among other things, the environment the "organizational man" finds comfortahle is the least productive of ideas). But Pelz's subjects (scientists) may be dif¬ ferent from programmers. Again, further research is advisable. -18- IV. CONCLUSIONS A niimber of exploratory stvidies were made to determine what factors might be related to success as a programmer. The findings were; (1) Both intelligence and motivation -- more notably, motiva¬ tion -- are closely related to classroom performance in programmer training. (2) Both intelligence and classroom performance are related to on-the-job ratings by supervisors. The stronger relationship of the two is that between classroom perfonnance and supervisor ratings. (3) Programmers have interests that clearly distinguish them from the lay population. A tentative Kuder vocational preference scale has been developed. Research in programming promises to be rewarding. The study sxjggests four areas: (a) predictive characteristics of programmers (particularly non-cognitive measures), (b) supervisory qualities, (c) the working environment, and (d) training.