Skip to main content

Full text of "DTIC AD0283796: FACTORS IN SELECTING AND TRAINING PROGRAMMERS"

See other formats


UNCLASSIFIED 


r 



lieoAoiutced. 
hu the 


ARMED SERVICES TECHNICAL INFORMATION AGENCY 
ARLINGTON HALL STATION 
ARLINGTON 12, VIRGINIA 



UNCLASSIFIED 








NOTICE: When government or other drawings, speci¬ 
fications or other data are used for any purpose 
other than in connection with a definitely related 
government procurement operation, the U. S. 

Government thereby incurs no responsibility, nor any 
obligation whatsoever; and the fact that the Govern¬ 
ment may have formulated, furnished, or in any way 
supplied the said drawings, specifications, or other 
data is not to be regarded by implication or other¬ 
wise as in any manner licensing the holder or any 
other person or corporation, or conveying any ri^ts 
or permission to manufacture, use or sell any 
patented invention that may in any way be related 
thereto. 




ME MORANDUM 
RM-3245-PR 

AUGUST 1962 






(Zil'J 


cs. 

l-v. ■ 

c:) 

C'- 


cri 



«cX, '■ 

I- 

c,:’- ''-X 


FACTORS IN SELECTING 
AND TRAINING PROGRAMMERS 

Anders Sweetland 


PREPARED FOR: 


A Q T I 

/ I V ■ f 


UNITED STATES AIR FORCE PROJECT RAND 



7^ 


RiinD 




SANTA MONICA • CALIFORNIA- 


283 796 






MEMORANDUM 
RM-3245-PR 

AUGUST 1962 


FACTORS IN SELECTING 
AND TRAINING PROGRAMMERS 

Anders Sweetland 


Th is research is s[)oiisored hy the I'nited States Air Force under I’roject HAND —Con¬ 
tract No. AF 49(6dd)-T()0 inonitorcd l>y the Directorate of Development Planning, 
Deputy Chief of Stall. Heseari h and Technology, Hq I'SAF. Views or conclusions con¬ 
tained in this Memorandum should not he interpreted as representing the olTicial opinion 
or policy of the United States Air Force. Permission to quote from or reproduce portions 
of this Memorandum must be obtained from The RAND Corporation. 








-iii- 


PREFACE 


A request from the Coii5)troller' s Office, SEAM, for assistance 
with the selection and training of programmers used in Electronic 
Data Processing work, provoked a reconsideration of a number of 
findings the author had uncovered in the programmer training program 
at the System Development Coiporation. The findings should he of 
interest to those groups or organizations who enploy programmers ,•>< 
or are considering the development of an electronic computer instal¬ 
lation. 

Although the results presented in the Memorandum are preliminary, 
they do point both to the paucity of conclusive research on this 
inportant occupation and to the desirability of further studies 
aimed at the selection and training of programmers for all types of 
EDPE installations. 


*It is necessary to understand that this study is concerned 
with programmers. Programmers are people who control the behavior 
of Electronic Data Processing Equipment (EDPE). Their activities 
are quite different from those people who control the behavior ol 
Electric Accounting Machinery (EAM). (Also called Punched Card 
Accounting tiachinery ... PCAM.) 





-V- 


SUMMARY 

This study atten^jts to determine some of the factors related to 
the selection and training of computer programmers. 

Section I describes the evaluation of nine classes of programmer 
trainees according to their intelligence, motivation, and classroom 
performance. Supervisors' ratings were also obtained as a follow-up 
study. The findings show that both intelligence and motivation, par¬ 
ticularly motivation, are good predictors of classroom performance. 
Intelligence is also a predictor of supervisors’ ratings, but not as 
good a one as classroom showing. (The data did not permit testing 
the relationship between motivation and supervisors’ ratings.) 

The results described in Sec. I suggested that it would be 
profitable to explore non-cognitive measures. Section II deals with 
a study of one such measure: the vocational interest inventory. 

The investigation showed that programmers have interests that clearly 
distinguish them from the lay population. As a result, a scoring key 
for the Kuder Vocational Preference Record was developed. It is 
desirable to en^jhaslze that this key should not be considered a 
final product, however, but rather as an illustration that such a 
key is feasible. 

Section III discusses the potential fruitfulness of research in 
programmer characteristics, interests, and aptitudes; it sxiggests four 
areas for such research: (l) the organismic factors, with emphasis 
on characteristics other than intelligence, ( 2 ) programmer svipervi- 
sors, (3) training, and (h) the working milieu. 





CONTENTS 

PREFACE . lil 

SUMMARY . V 

Section 

I. RELATIONSHIPS AMONG TRAINING, INTELLIGENCE, AND 

MOTIVATION . 1 

II. AN INVESTIGATION OF A NON-COGNITIVE MEASURE . 8 

III. SUGGESTIONS FOR FURTHER RESEARCH . I 5 

IV. CONCLUSIONS . I 8 












- 1 - 


I. RELATIONSHIPS AMONG TRAINEES. INTELLIGENCE AND MOTIVATION 

About 195^> the Numerical Analysis Department of the RAND 
Corporation felt the need for more formal selection procedures to 
meet their burgeoning demand for computer programmers. They approached 
the System Research Laboratory now the System Development Corporation 
(SDC) for assistance. 

As a first step, members of the department underwent a battery 
of tests and were also ranked (subjectively) for programming ability. 
The tests were correlated with the rankings, and a multiple regres¬ 
sion analysis made. (The multiple correlation coefficient was 0.59)- 
The analysis yielded fo\ir tests that accounted for most of the 
explainable variance. 

Thurstone Primary Mental Abilities (PMA) Test: 

1. Verbal 

2. Reasoning 

3. Spatial 

Thurstone Temperament Schedule: 

4 E (emotional stability). 

Both RAND and SDC have used this set to date, with minor 
variations. Both have dropped the E scale (which is awkward to 
explain and to administer). Both override the percentiles cut¬ 
offs if the applicant has compensating qualities (such as extensive 
background in mathematics, or programming experience). SDC requires 
that applicants have had at least one course in calculus. 

Since the major components of most intelligence tests are 
measurements in the verbal-reasoning axea, it is a I'easonable 


- 2 - 


conclusion that the two companies are limiting their hires to 
approximtely the upper 2 to 5 per cent of the population in general 
intelligence (roughly, those with IQ's of 120 and up). 

Although the use of the PMA is a little hard on t he recruiters, 
the system works well: fewer than 5 per cent of terminations are due 
to lack of ability. As will be shown, however, intelligence is not 
the sole determinant of programming success. 

Shortly after setting up the training program at SDC, we dis¬ 
covered that the PMA scores could not be used for counseling trainees; 
there was insufficient spread (as a result of the two 90 th-percentile 
cut-offs used in hiring). It was impossible to determine the relative 
level of Intelligence of the individual student. We desired a test 
that would: 

(1) Measure in the same dimensions as the PMA; 

(2) Distinguish among trainees; and 

(3) Be easy to administer and score. 

The Otis-ttlgher Examination, Form D, met all three requirements. 

A factor analysis (coupled with other tests) showed that it measured 
the same dimensions. A reduction in testing time to 20 minutes 
yielded a spread from 25 to 75 in the raw scores. It could be 
administered to a group in less than 25 minutes, and scoring took 
less than a minute for each person tested (see Fig. l). The measure 
of intelligence (unless stated otherwise) in the remainder of this 
study is the raw score obtained on the Otis Form D taken under a 
20 -minute time limit. 

The programming classes were composed of approximately 20 students 
each. Each class ran eight hours a day and lasted eight weeks. As 




Fig- 1 — Scattergram of Otis Scores (20-Minute Time Limit) 









each unit of work was completed (lasting 3 to 5 days), the students 
were tested for knowledge in that unit. A constant program of item 
analysis was carried out. After the first four or five classes, the 
test items stabilized at the 50 -per-cent level; i.e., each item 
was answered correctly by half the class. In the comparisons that 
follow, the measures of performance are based on the total scores 
for the eight-week training period. These vill be either the raw 
scores, or the raw scores converted to rank-orders. 

At the time, we were working on some characteristics of human 
motivation, and decided to include part of this work in the present 
study. Two estimates of motivation were used (with motivation defined 
here as a desire to lefitrn programming): 

(1) Estimates from instructors . Each class had two instructors. 
At the end of the course, each ranked the students independent!;,''. 

The lists were then compared and discrepancies resolved by discussion, 
which resulted in a third ranking. This third ranking was one of 
the measures used in the comparisons• 

(2) Estimates by peers . Each student ranked the other nxjmbers 
of the class (omitting himself). The rankings were first tested for 
stability, (using Kendall's Coefficient of Concordance*) and the 
composite rankings were rank-ordered. This composite ranking is 

the measure used in the comparisons. Because the peer estimates 
proved to be much more stable than the instructor estimates, we 

* 

M. G. Kendall, Rank Correlation Methods , Chsirles Griffen & 

Co., Ltd., London, 1955- 



-5- 


Table 1 

CX)RREIATION BETWEEN UnELTIGENCE AND CLASS PEePORMAECE 
AMD MOTIVATION AND CLASS PERFORMANCE 


Class 

Intelligence: 
Otis & Grades 

Motivation 

Inst & 
Grades 

Peer & 
Grades 

3 

0.31 

0.34 


4 

0.30 

0.66 


5 

0.59 

0.82 


6 

0.37 

0.49 

0.71 

7 

0.23 

0.46 

0.82 

8 

0.44 


0.74 

9 

0.30 


0.67 

10 

0.44 

« 

0.57 

11 

0.60 


0.61 

Mean 

0.40 


(?»$9 


NOTE: Instructor and peer rank¬ 
ings were used ais estimation of mo¬ 
tivation. Kendall's Tau was used as 
the correlation meastire. (Since the 
experiment began with Class 3^ 1 and 
2 are not included.) A Tau of 0.27 is 
significant at the 0.05 level for 
N = 20, the average class size, 
finally used them exclusively. 

The relationship among intelligence, motivation, and class grades 
is shown in Table 1. In this compeirison, the Otis scores and class 
grades were converted to rank orders. Kenall's Tau was used as the 
statistical test. (To those more accustomed to the conventional Pearson 
product-moment, a Tau of O. 3 O is about the same as a Pearson product- 
moment of 0 . 45 ). 


Since the time these data were collected, a number of studies 
relating intelligence and performance have been compiled. These are 
summarized in aui excellent paper by Dallis K. Perry, Computer Progrft»nm<»T- 
Selection Testing , Field Note 6371, System Development Corporation, 

Santa Monica, California, MEirch, 1962. 

** 

Kendall, 0 £. clt . 











- 6 - 


The interesting peurt of these data is the ability of motivation 
scores to predict grades. In eveiy instance, the estimations of 
motivation predict grades better than do the fonnal measures of 
intelligence. It thus appears that it would be profitable to investi¬ 
gate characteristics other than intel l igence, a conclusion bolstered 
by the following observations. 

At one time, as part of a salary review, each supervisor ranked 
his staff according to their "value to the company" and, this done, 
was asked to convert his ranking to a company-wide percentile ("Of 
all the programmers in the conpany, where would you place this 
person, percenteige-wise?" ). 

As a part of the quality improvement of the training program, we 
collected these supervisors* ratings. Admittedly, they are imprecise. 
They came from several points in time: sometimes shortly after the 
programmer was on the Job, sometimes after he had been there several 
months. The supervisory experience was equally varied, nor was it 
possible to get 100-per-cent samples. Thus the supervisors' ratings 
lack the nicety of laboratory data, but at least they are real-world 
measure s. 

We matched the ratings with the Otis scores and training scores 
and correlated the samples (using Pearson product-moment). Since 
each sample (N = 50) is a mixture of several training classes, no 
direct conparlson can be made with the previous data. See Table 2. 

Two conclusions appear warranted: 

(l) A student's showing during his training is important to an 
organization hiring programmers; it predicts his ratings by immediate 


-7- 


supervlsors. This fact in turn implies that the training program per se 
is (or should be) a major concern of the organization. 

(2) Programmer selection might be improved by investigating areas 
other them intelligence. 

The Otis test does a reasonably good Job of predicting class¬ 
room showing; and the classroom is a surprisingly good predictor of 
the supervisor's rating. 'Ibe Otis test does not predict the super¬ 
visors' ratings very well, however. 

Table 2 

CORREIATIONS AMONG OTIS SCORES, TRAINING 
SCORES, AND SUPERVISORS' RATINGS 


Otls- 

Grades 

Grades- 
Ratings 

Otis 

Ratings 

0.46 

0.66 

0.26 

0.40 

0.42 

0.15 

0.25 

0.48 

0.01 

0.49 

0.73 

0.29 

0.47 

0.32 

0.27 

0.27 

0.32 

- 0.11 

o. 4 o 

0.54 

0.29 

0 . 4 l 

0.28 

0.09 

Mean 0.39 

0.’'-7 

0.16 


NOTE: A correlation of 0.27 is 
nificant at the 0.05 level for N = 50. 


- 8 - 


II. AM INVESTIGATION OF A NON-COGNITIVE MEASURE 

Section I suggested that exploration of the-non-cognitive areas 
might improve the selection of potential programmers. During the 
interview, the recruiter faces one central question: whether to hire 
or not; to answer it, he needs objective measures that have predicted 
success in programming. Intuition does not help. F„r example, one 
would intuitively surmise that an applicant's mathematical background 
should foretell his success in programming. Experience proves other¬ 
wise. A sample of 40 cases was used as a test, in which semester 
hours of mathematics (which ranged frcm 6 to to) were correlated with 
class grades. The correlation was -O.I 7 (not significant). 

Vocational backgromd was equally unfruitful. So was the amoimt 
of college that applicants had had: some of the poorest students 
were Ph.D's, but so were some of the best. As in most professions, 
there were few outstanding females, but females (again as in most 
professions) were similarly rare at the low end of the continuum. 

Other than the intelligence test, our fictive recruiter has little 
to help him at present. 

These negative considerations lead to the following rationale. 

Most vocational guidance programs encompass three areas of measure¬ 
ment, two of which have been considered: (l) ability, (2) achievement, 
( 3 ) interest. The intelligence test measures ability and does 
nredict success in programming. The simple and obvious achievement 
measxu:es (amoimt of college, amount of mathematics, etc.) yielded 




- 9 - 


little. Finally, the area of interest-measurement has not been 
explored at all. It is an appealing one for two reasons: 

(1) A vocational interest score is believed to be a low-key 
measure of motivation; and 

(2) The data of Sec. I indicate that motivation plays a major 
role in success in programming. 

The following is a description of an exploratory study in 
vocational interest testing. The Kuder occupational preference 
record was given to 100 programmer trainees. Since trainees are not 
professional programmers, we had misgivings about using these data 
to establish a preference profile; however, it is comparatively easy 
to get data from trainees and very difficult to get data from pro¬ 
fessionals. Consequently, our first question was: Will trainees 
reasonably resemble professionals in their responses to the KvuierV 

The hundred trainees were split randomly into two groups. A 
Kuder key was established by standard procedures. (Essentially the 
procedure was to ask whether the proportion of the experimental group 
who answered an item in a specific manner differed significantly from 
the nomnative group proportion. If so, the item was incorporated in 
the scale. ) In this st\idy, an item had to distinguish (from the 
normative group) at the 5“Per-cent level in both experimental groups 
to be Included. 

This procedure established a scoring key for trainees. The 
next question was: Do professional programmers answer the K\ider in 

There is one unexplored area, however: success in medical school 
has been predicted by combining college grades in a specific set of 
related subjects. It may be that combinations of grades would have 
similar predictive value for programming success. 


- 10 - 


the SEune way trainees do? The answer sheets of the 100 students and 
of 30 BAND programmers were scored with the key constructed from the 
trainees' answers. In addition, 100 answer sheets from the general 
population were obtained and were scored with the same key after 
being split into two random groups. The results of this manipulation 
are shown in Figure 2. 

The reader can see from Fig. 2 that trainees and professionals 
answer the Kuder in essentially the same manner, and that their 
interests (as measured by the Kuder) are quite different from those 
of the general population. 

Following this happy discovery, we made several trial-and-error 
attempts to find a scoring key that would yield the least overlap 
among the three samples of programmers and the two samples from the 
general population. The resisting key was obtained by isolating 
those items that distinguished programmers from the lay population 
in all three of the experimental groups (i.e., to be included, an 
item had to distinguish in each of the three separate samples). The 
emswer sheets were rescored with this key. It is obvious that pro- 
grEunmers have interests different from those of the lay population. 
(See Fig. 3 for resvilts.) 

Two precautions are in order: first, although the area of 
vocational preference appears to be promising, there has been no 
validation for this key; and although programmers can be distin¬ 
guished from the general population in terms of their interest, it 
remains to be demonstrated whether these interests are related to 
success in programming. (For example, if we partial out intelligence. 



- 11 - 



Fig. 2 — Kuder Scores, First Trial: Scattergrams of 
Three Programmer and Two General Population Samples 






- 12 - 


100 SDC + 30 RAND 

100 Norm 

50 



49 



48 



47 



46 



45 

// 


44 

1 


43 



42 

THt 


41 

HU 


40 

III 


39 

mt IHJ ifi 

4 

38 

mm If 


37 

mm mm 


36 

numii 


35 

m m Iff 

/ 

34 

rHl HU 

// 

33 

HU tru rm 


32 

rm m 

m 

31 

HU 

HH 

30 

H 

7m TtM 1 

29 

HH 

HH / 

28 

III 

ITH / 

27 


HH 

26 

/ 

HH HH 

25 


fUt H 

24 


mi H 

23 


HH HH 

22 


HH 

21 


rm HH 

20 


HU ! 

19 


/ 

18 


H 

17 


H/ 

16 


/ 

15 




M - 35.684 M - 25.410 

O'- 3.646 O'- 4.338 


Fig. 3 — Final Kuder Key Scoring, Comparing 
130 Programmers with 100 General Population 





-13- 


will the Interest scores be related to success In training, success 
on the Job, or continxiatlon in the profession of programming?) 

Until the key has been validated, it is desirable to follow the 
more conservative approach used in the VA program (Public Law #l6 
and Public Law following World War II. In the VA program, 

unless a person showed a career interest above the 751 h percentile 
of the general populatlcxi, it was deemed that his interests were 
not crystallized enough to warrant his planning for that career and 
he was encouraged to explore further. (We may note in passing that 
a 75-per-cent cut-off of the general population would have cut off 
the lower 7 per cent of the programmers used in this study.) 

Second, unlike intelligence (which is fixed at conception), 
interests are malleable. It is generally true that a person's 
vocational interests tend to stabilize when he is about twenty-five, 
but this is largely a cviltural happenstance. A person can become 
interested in a new profession at the age of 50. This is smother 
way of seiying that vocational guidance, based on interest testing, 
should operate only in the crudest of dimensions: a man with an 
interest in science and mathematics and none in social services 
should presumably be steered toward the engineering and related 
professions, where he is more likely to find a stimuJ-ating occupa¬ 
tion, and away from the social-service professions. Because interests 
are learned, however, it would be incorrect to assume that such a 
man cannot become interested in social services. 

To summarize this Section: Programmers ajxpear to have a set 
of homogeneous interests that distinguish them from the general 






population, and the selection of programmers mi^t be improved by measuring 
these interests. Interest scores should be used conservatively, however, 
until interest is shown to be a valid predictor of programming success. 

If interest measures are to be used, it is suggested that the upper 25 per 
cent of the general population be used as a recruiting cut-off point. 


-15- 


III. SUGGESTIONS FOR FURTHER RESEARCH 

The explosive growth of computer technology in the past ten 
years indicates that the profession of programming warrants consider¬ 
able exploration. Computer installations are expensive, and what we 
get out of them is highly related to the talents of the programming 
staff. It is worthwhile to begin exploring the dimensions of pro¬ 
gramming talent. 

The programming activities of most large computer installations 
fall into two broad categories: 

(1) Service functions . These are the bread-and-butter jobs: 
payroll, billing, insurance premium computations. Essentially, ex¬ 
ternal needs define the programmer's actions. The Jobs generally 
make considerable demands on his ingenuity, but rarely necessitate 
inspired, creative imagination. 

(2) Research in programming . In contrast to the bread-and- 
butter tasks, basic research is another world. In it, people try 
to get the computer to do things Turing never had in mind, such as 
translations of language, learning, concept formation, problem¬ 
solving (in the psychological sense), and abstract writing. Here 
imagination is a sine qua non . 

Intuitively, we would surmise that these two types of activities 
demand two different types of programmers, although we do not know 
what their distinguishing qualities may be. (We may be wrong, however: 
perhaps the same programmer can double in both types of activities.) 

In one instance, the program is a means to an end (as statistics are 


-16- 


used in psychological experiments); in the other the program is an 
end in itself (e.g., the development of linear programming). Research 
on this question is advisable. 

A second area for exploration is determination of the qualities 
of good programmer supervisors. From a company's point of view, good 
supervision is critical; but how does the recruiter respond to the 
request, "Of the ten programmer trainees you recruit, include two 
who are potential supervisors?" More basically, what qualities (if 
any) distinguish good programmers from good supervisors? 

A third area for research is training. One question is how old 
(or young) a person should be when he begins to learn programming. 

Some of the work on concept formation suggested that the ability to 
form concepts is independent of chronological age but directly related 
to mental age. The same may be true of programming. Extrapolating 
the Otis results suggests that the brighter (upper 5 per cent) juniors 
and seniors in high school could be trained in programming and benefit 
by the process, but that it would be an unusual high school freshman 
who could do as well. Psychologists recognize this as a variation of 
the maturation-versus-learning problem. 

Finally, there is the comparatively unexplored area of the work¬ 
ing milieu. If it is true, as our data imply, that the programmer is 

a different sub-species, perhaps he needs a special kind of environ- 

* 

ment in order to be productive. The work of Pelz on the productive 

*1). Pelz, "Some Social Factors Related to Performance in a 
Research Organization", Adm. Scl. Quart ., Vol. 1, No. 3, December, 

1956 , pp. 310 - 325 . 


-17- 


milieu suggests that the scientifically productive working environment 
differs dramatically from what one might guess (among other things, the 
environment the "organizational man" finds comfortahle is the least 
productive of ideas). But Pelz's subjects (scientists) may be dif¬ 
ferent from programmers. Again, further research is advisable. 


-18- 


IV. CONCLUSIONS 

A niimber of exploratory stvidies were made to determine what 
factors might be related to success as a programmer. The findings 
were; 

(1) Both intelligence and motivation -- more notably, motiva¬ 
tion -- are closely related to classroom performance in programmer 
training. 

(2) Both intelligence and classroom performance are related to 
on-the-job ratings by supervisors. The stronger relationship of 

the two is that between classroom perfonnance and supervisor ratings. 

(3) Programmers have interests that clearly distinguish them 
from the lay population. A tentative Kuder vocational preference 
scale has been developed. 

Research in programming promises to be rewarding. The study 
sxjggests four areas: (a) predictive characteristics of programmers 
(particularly non-cognitive measures), (b) supervisory qualities, 

(c) the working environment, and (d) training.