ED 064 353
TM 001 521
Huberty, Carl J. ; And Others
An Evaluation System for a Psychoe(5ucational
Treatment Program for Fmotionally Disturbed
27p. ; Paper presented at the annual meeting of the
American Educational Research Association (Chicago,
Illinois, April 1972)
^Behavioral Objectives; =<« Emotionally Disturbed
Children; ^Evaluation Methods; ^Measurement
Instruments; ^Psychoeducat ional Processes; Rating
A general description of an overall evaluation system
which is being implemented in a center for emotionally disturbed
children is presented. The system is based upon three types of
activities; planning, monitoring, and appraising. The application of
the system to the evaluation of direct services to children is
outlinai* The evaluation plan for the child treatment program
involves five phases; intake, staffing, monitoring, termination, and
tracking. Three periodic measurement instruments used during the
monitoring process are discussed: a clinical behavioral scale
completed by a psychologist; a behaviorally ba'^ed instrument
completed by trained evaluators; and a rating form completed jointly
by a monitor and therapist (s) . (Author/DB)
TM 001 5 21 Ed 064553
U S. DEPARTMENT Ot= HEALTH.
EDUCATION A WELFARE
OFFICE OF EDUCATION
THIS DOCUMENT HAS SEEN REPRO-
DUCED EXACTLY AS RECEIVED FROM
THE PERSON OR ORGANIZATION ORIG
INATING IT POINTS OF VIEW OR OPIN-
IONS STATED DO NOT NECESSARILY
REPRESENT OFFICIAL OFFICE OF EDU
CATION POSITION OR POLICY
AN EVALUATION SYSTEM FOR A PSYCHOEDUCATIONAL
TREATMENT PROGRAM FOR EMOTIONALLY DISTURBED CHILDREN*
Carl J. Huberty
University of Georgia
John P . Quirk *
William W. Swan
* Paper presented at AERA Annual Meeting, Division D, Chicago, April, 1972.
t* Now at Indiana State College, Indiana, Pennsylvania
FILMED FROM BEST AVAILABLE COPY
All Lv.'u uat a ona 1. Syst^am for a Psychoeducational
Treatment Program for Emotionally Disturbed Children
A general description of an overall evaluation system which is
being implemented in a center for emotionally disturbed children is
presented. The systeni is based upon three types of activities: planning,
monitoring, and appraising. It is pointed out that in such a three-
pronged model these activities are neither independent nor mutually ex-
clusive; they are not only compatible, but mutually supportive. It is
the interrelations of the three types of activities which produce the end
product. Once the initial planning is completed, the model affords
reassessing, modifying strategies, and reprogramming whenever desirable.
Following decisions of reprogramming, the evaluative cycle repeats it-
self: planning, monitoring, and appraising. It is for this purpose that
a well developed information exchange system within the center is needed.
Such evaluative procedures make it possible to advantageously integrate
data collection into the decision-making process.
The goals of the evaluation team are; 1) to assist in expressing
questions to be answered and information to bs obtained, 2) to collect
the necessary information, and 3) to prepare the collected information
in a form useful for decision makers for assessing decision alternatives.
The Information to be used in each component program is in the form of
data that provide descriptions and judgments of anything which feeds in-
to the progi'am (antecedents), happens during it v -cransactions) , and re-
sults from it (outcomes), along with the contingencies among these. The
antecedents constitute a major contribution to the planning and develop-
ment of the evaluation strategy(ies) to be subsequently employed. It is
a function of the evaluation team to relate the transactions to the ob-
jectives and processes of each component. The concern with the output
data is one of devising performance criteria, relating these data to
the other two types of data, and formulating ‘decisions regarding worth
and attainment of component objectives.
The somewhat detailed application of the system to the evaluation
of direct services to children is outlined. It is emphasized that the im-
portant prerequisites of an evaluation svstem of a chi.M treatment nrogram arc
that the system be easily implemented i cui usr>^u’ .
The evaluation plan for the child treatment program involves' five
phases: intake, staffing, monitoring, termination, and tracking. Through
a problem check list, a langua^ common to individuals of varying back-
grounds, from parent to psychiatrist, is used through the first two phases.
A second language, closely allied to that of the problem check list, in the
form of a list of treatment objectives, provides a commonality the Rutland
Center professionals upon which a meaningful monitoring process has been
dovsloped. Three periodic moasiirement instruments used during the
monitoring process are discussed* One instrument, a Ciinical behavioral
scale completed by a psychologist, measures the qualitative aspects of
behavior; a second instrument, behaviorally based and completed by
trained evaluators, measures the quantitative aspects of behavior; and a ^
third instrument, a rating form jointly completed by a monitor and thern^
pist(s), measures both aspects of behavior. All Liiree iristrumcnts wore
developed from the tv;o common langungn,’. mentioned previouslv.
It was discussed how the evaluation system provides foi*’. a- periodic
feedback of information which is useful in supporting decisions regarding
the individual child and the treatment program. Such information, along
with that obtained at intake, aids in deciding when the termination pro-
cess should begin. It was discussed how further evaluation will be made
during termination and after direct Center treatment ends.
an EVALUATlOh’ SY:;TEM irOR A PSYCHOEDUCATIONAE
TREATMENT EROCRAM FOR EMOTIONA'.LY DISTURBED CHILDREN
The Evaluation System: An Overview
A current thrust in the efforts at Rutland Center' is the develop-
ment of an evaluation system. This system is considered an integral,
part of the overall project rather than an adjunct to it, and the eval-
uation personnel have taken, and are taking, an active role in the plan-
ning, monitoring, and appraising phases oi all Center operations. Be-
cause of this total involvement the success of the evaluation system is
dependent upon a well developed system of information exchange which en-
hances feedback and communication. The involvement of the evaluation
team in the total project and its participation in the exchange of in-
formation are depicted in Figure 1. Note that the evaluation team is
expected to provide evaluative services (in the form of planning, moni-
toring, and appraising) to each of four components: demonstration and
dissemination, training, service-to-children , and service-to-parents .
(Although data are collected for the purpose of demonstration and dis-
semination, the following comments in this section generally pertain to
the other three components . )
Insert Figure 1 about here
The goals of the evaluation team are: 1) to assist in expressing
questions to be answered and information to be obtained, 2) to collect
the necessary Information, and 3) to prepare the collected information
in d form usi-iai ior decasioi. inaxers U.r asS' :'.sing '.!eoi.sion alternatives*
The informati' n lo used • n each component progr-am is in thQa^jform of
data that provide d-^scrJ pt lens and judgments of anything which feeds into
the program (anteceden- ■) , happens dur-ing it ( i rarisactions ) , and resu-its
from it (outc:omes)j along wrth the contingencie' • among these (see Stake,
1967). The antecedents include such inputs as trainee, child, and parent
characteristics, referral data, environmental factors, and the psycho-
educational curriculum and techniques. These inputs constitute a major
contribution to the planning and development of the evaluation strategy( i- s )
to be subsequently employed. Involved in the transactions are the pro-
cesses and interactions witnin and among learning or training activities,
individuals, and materials. It is a function of the ovaluaticui team to
relate such data to the objectives and processes of each component. The
outputs pertain to the individual client, to the home , and co the Center.
The concern with the output data is one of devising per-tormance criteria,
relating these data to the other two types of data, and supporting de-
cisions regarding attainment of component objectives, need for treatment
modification, need for reprogramming and recycling and readiness for term!
nation of treatment.
The various functions and roles of the evaluation team within the
framework of the Cenlei' are outlined in Figure 2. It is important to
note that in the three-pronged model these three types of evaluation activities
Insert Figure 2 about here
ai’e neither Independent nor mutually exclusive; they are not only compatible,
but mutually supportive. As with the three kinds of data used for the evalua-
i7. I p . u> : ‘ v'P - / ’ ! • <- * • ■. ; ' ‘ ^ • JC * ‘ ^
w nr / i* •• -.'.‘ i- • 'v - • < " T ■» :i ' ’■
. ^ ■ rr.. . » r • t
. r. , -r- . • i w i ■: iV^r!iv.. nfnr ^ ' vh‘ -.o.^ . r-n'' r-‘’'r' 0 '-'r.t'.it' . n -
c/"' •.'. "iS i . I'o. ('■■?■* '..sVvP.j of re jf : .’z'. .r:":.i p'' . the ev '.vo
■•'''r.eatc: i :o j t. * i ■J.vinriiPP’ r- >ni ’■-orin'"- , ai>( 1.'-. i-''. > r. ’■ ■ ■ ••
ac a v;eli deve.-opc .i u' . o '''ration exe'-a- •'
svst'-’'’' T. f >n Pfsni
edsd. Such ev*axU'-t vo nr’-.'ico'i.ur'S^- make : a tc. . v-.zit.a'''eou . '•
intej^': :zte data eoilecc'on into the de'^-ision -n-'.
, , ■ -rf-s.s .
heeause o? theoneticaj. iimvtati ons , and JtJi’ ni'af i aip’ ethical
reasonr.. (especially with regard to the service- l o-chJ Ldr.-en fiorp.^r-enL) ,
the evaluation plans do not caiJ. tor a con-paraz. : ve ...isczs'jment C" trear
merits or curricula. That is. the system does nor iriciude vzhai hs found
in. tyT'icci 'research' or ‘experimental" se-^unz>;- r.ai'eJ.v, random samples,
constant ' treatment ‘ controlled variables, anc’ (.ornoarison or control
groups. Rathe'*^ . the concerfz is vzith detai.Led tier. cr i ot ions and. obaer .^riop-.
of individuals or small o nuses. The posiiion is sinv'Jar to tha^ of
Cronhach (1963)- the ain to compare one nrograni v.iic another shouLd not
dornin-'c e plans i cr evaiua toni evaJ.uation shuiuid >e tjr.tpari ’v conce''’nt-..d
V 71 th the ef.toi:>'.s of thei program under sf ud'?. t ^r, j ^ind Center efiert is
addressed to the question, ’’What changes < ao be observed in a certain .'cizvi
.individual v/hic'n can be attributed to an Involvam.em in a cert.iji- .k.inc c' t;
gram intervention?" Come time ago, ’ iibcrckv (j9S9 . p. 328) poir'tec- out
It has yet to be demonstrated that control nrouos m psychotherapy re-
search have a moi-e than very limited usefulness.’* The literature ■ -.c t.ie
past decade has not produced much evidence to the contrai’y •
The Rutland Center evaluation methodology is not necessarily de-
signed to yield universally valid information: the focus is on thes^ par-
ticular treatment processes, integral parts of this psychoeducational model.
The emphasis in the evaluation program may be likened to a current emphasis
(controversy?) in educational measurement namely* that of criterion-
referenced measures. Rather than comparing the performance of individuals—
trainees, children, parents — in the Rutland program with other individuals
(norm-referencing), criteria are being established for each individual,
thus enabling the individual’s progi’ess to be assessed relative to himself.
(This does not, of course, preclude the use of norm-referenced measures ob-
tained from ’’standardized” tests to yield input data.) These criteria for
attaining objectives are usually not determined until after the individual
receiving services has entered the program and some assessment has been made.
And the decision of whether or not an individual has attained a criterion
established for him is based upon as much objective information as possible
(test results, systematic observation, rating forms, etc.), supplemented by
whatever clinical judgment is deemed pertinent. Such decisions are made, of
course following discussions involving an evaluator, a teacher, a psycholo-
gist, a monitor, and anyone else who may be familiar with the individual.
The success of such an evaluation methodology is highly dependent upon
explicit statements of the goals and objectives of each of the project
components . The inputs , transactions , and outputs must dii’ectly relate
to the general objectives of each component as well as to specific objectives
associated with the individual trainee, child or parent. The inportance
and role of the objectives are clearly reflected in the three-pronged
model (planning, monitoring, appraising) discussed previously. The
emphasis is on (measurable) objectives as guidelines for action, and on
meaningful observation and description in assessing an individual’s pro-
gress, or lack of it.
Most of the evaluation effort extended to data has been focused on
the treatment progx'am for the service-to-children component . Considerable
work has been done in planning for the evaluation of the effect of the
Developmental Therapy^program on four classes of preschool emotionally
disturbed children at Rutland Center. The remainder of this paper dis-
cusses the application of the evaluation system outlined above to a
method of treatment designed to ameliorate the child’s symptoms and to pro-
duce gains in those areas which are most debilitating to his functioning.
Evaluation of Service to Children
For an evaluation system to be employ®<i iri a treatment program it
must not only be empirically sound but, more importantly , it must in
the long run be useful for clinical practice. To be clinically useful,
an evaluation system must be intimately tied to the philosophy and under-
lying theory upon which the treatment program is based. This has been
particularly difficult for traditional treatment programs which focus
exclusively on broad hypothetical constructs related to psychodynamics.
The emphasis at Rutland Center, however, is on problem behaviors manifested,
or perceived, in the home and/or the school. Having a problem behavior
orientation instead of a mental illness framework has made it possible
to develop specific behavioral objectives for treatment planning and for
measurement purposes. Recognizing also that qualitative aspects of be-
havior are important, provision has been made in the evaluation system for
the measurement of these aspects.
FILMED FROM BEST AVAILABLE COPY
In addition to the need for consonance with a theoretical base,
.iii cvaluaiion system must be composed of procedures that can blend
. Into the everyday, functioning of a treatment center. Any syste’-
triai; takes an inordinate amount ref fort and does hof’fa^'
-iitate the treatment function will soon be discarded. Practicing edu-
caiors , psychologists, and social workers need evaluation procedures
wiih which they can be comfortable and which help them be more effective
in dealing with children’s problems.
Thus, to be affective, an evaluation system must be built into the
treatment program itself. Objective delineation of problems, setting
treatment goals, periodic assessment of progress, and the utilization
of objective or quasi-objective^ data for making treatment decisions
should be not only qualities of a useful evaluation system but also
necessary characteristics of any productive treatment program for children.
The general goal of the servlce-to-chlldren component is; to pro-
v'ide psychoeducational treatment experiences to referred children so as
to enable them to better cope with their home and school environments,
-aruirable outcome objectives’ f dr the' >childre'tt’’^nv61ve decreasing the
•: amber and/or severity of blhd’VibrSi''-^ and improving appropriate
■akiJl; .in curriculum areas of the psychoeducational process.
tion system is to assist in the planning and maintenance of the treatment
program for children. However, since the philosophy of treatment here is not
strictly behavioristic, a potential difficulty existed at the outset.
It was felt that the objectives must reflect both the developmental aspects
Structuring of the Treatment Program
of the treatment model and the qualitative aspects of behavior, and at the
same time maintain a sufficiently behavioral orientation to allow for
somewhat objective and reliable measurement. Extreme ..specificity in the
st<^tement -of -objectives would have -had a limiting -effect .on the. psycho-
educational therapists, while over-generalization would have made the ob-
jectives difficult .to assess.
From this demand for a balanced approach , -the list -of representative
objectives^ resulted .. These objectives provide behavioral milestones
around which the treatment program of a chi Id .can be -planned and monitored.
The objectives .range - from. simple .attending and ..responding -behavior neces-
sary for any constructive child-environment interaction. to more complex
social skills ihch as those involving lel^dership behavior. They were de-
veloped around the.four.^ curriculum areas pf -Developmental -Therapy: be-
havior, communication, .socialization, .and school readiness . -An attempt
has been made to specify .the .hierarchical order in which these behavioral
ob j actives appear - in . the - developmental .process • This • list serves as a
common language. useful. for . the purpose of outlining measurement -procedures
and constructing data- collect ion instruments. This . commons 1 ty maximizes
communication among the various staff members involved in the periodic
The Evaluation Plan
The evaluation plan. for ..the service^to-^ohildren. component is viewed
as consisting of . five major phases which coincide with -the flow of diag-
nostic and therapeutic .procedures of the tipeatment progr%<' The phases
are intake, staffing, monitoring, termination, and tracking. F.ach phase
is directly supported by data collected and sumraarized by the evaluation
team (see Figure 3). The evaluation team assists in the delineation of
Insert Figure 3 about here
the child’s problems during intake and staffing, provides periodic feed-
back information necessary for . maintaining and adjusting the treatment
program, assists in speci^ing ..termination criteria, and obtains follow-
up information after direct Center treatment ends .
The evaluation ..and .monitoring . effort .begins with .the - initial contact
with parents and regular teacher and ends approximately one year after
the child has been. terminated -from the treatment program. . .Throughout the
diagnostic, staffing, .and -treatment . phases . of the .program. the evaluation
system yields important .informational feedback to .the. professional staff.
All of the professional .staff .members participate in the development of
procedures which provide the required data. These procedures are aimed
at increasing the. amount and usefulness of objective and quasi-objective
data employed in making clinical judgments.
Intake and - Staffing . .
Many multi-disciplinary treatment teams have found it difficult to
delineate problem areas to. the satisfaction of all involved. A common
language , which . facilitates .communication among educators , psychologists ,
psychiatrists, social workers, measurement personnel,.parents and regular
classroom teachers, .is essential if a .child.is to. receive maximum benefits
of a treatment program.. Provision for such a . common language rin the deli-
neation of children’s proiilem areas is made by the Referral Form Check
The RFCL is a composite of behavior problems abstracted from re-
ferral records accumulated over a two— year period. The treatment files
ware reviewed and all referral problems for preschool and primary
school children were listed. Over 200 behavior problems were recorded;
from this list -many were eliminated because of . duplication of problem
meaning. This .synthesis resulted in the check list, which is composed
of 54 behavior problems grouped within the four curriculum areas of
Development al. Therapy. - -A. review of. the literature (£.£. , Peterson and
Quay, 1967; Kooi and Schutz, 1965; Schrupp and Gjerde, 1953) Indicated
that the RFCL. contained -characteristics which .are -identical or parallel
to those that have bean previously . investigated. • A . five-^point rating
scale format , .ranging, from -’’High. Priority Problem’’ to ."Not a Problem
or Not Noticed" .was selected -because such a format . (a) provides a
range for detection .of behavioral change over .time, (b) .allows for re-
cognition of problems .perceived .by . adults as ..’’real'.’ adjustment problems ,
and (c) permits the incorporation of clinical inference in the judgment
Investigation of reliability of the RFCL is currently in progress .
Inter-observer . reliability . estimates have been obtained using an intra-
professional . group . C i.»^» , educators , psychologists , etc . ) orientation .
Initial results are encouraging. Using the coefficient suggested by
Ebel (1951), reliability estimates range from .46 to .76 across pro-
During the intake procedure , ratings ^bn the RFCL are obtained from
each staff member who is involved in the diagnostic process (educational
tester, psychologist, and psychiatrist). In addition, -RFCL's are coii^ieted
by the child^s parent(s) and regular classroom teacher. The multiple
perception of a single, pool . of pronlem behaviors -has ..been extremely help-
ful in facilitating the presentation ofa comprehensive picture of a
child during. staffing. .The .evaluation. team. collects .and summarizes the
data from. all. of the RFCL's completed. Subsequently, .at . staff ings this
information is summarized via a RFCL profile. bar graph and summary sheet.
The summation. of. perceived problems thus seems not only . to solidify
thinking as a staff,. but also to reduce. the need for ■ detailed diagnostic
reports from. each staff member. . Brief clinic staff -reports are given
which focus mainly on. the possible etiological factors that have been de-
rived through clinical judgment. Allowing for. multiple hypotheses in de-
termining the source of a child’s problem has proven invaluable for main-
taining a flexible. treatment approach.
(Pilot tesL-.-’.ng is planned for the utilization of this same RFCL for
the purpose of obtaining post-r treatment measures from parents and regular
classroom teachers for the detection of problem change, or change per-
ceived by the adults .involved. )
Other data are also obtained prior' to staffing. Tests measuring
such things as social behavior, perceptual-motor development, and academic
readiness are administered; intelligence and projective measures are also
To facilitate program planning and subsequent monitoring, the staffing
information -is recorded .on ..a three- columned treatment sheet. The first column
contains all the high priority problems. The second column contains the
suggested causative factors underlying the behavior problems. Tlie third
column outlines the treatment focus with specific suggestions for be-
havioral objectives needing emphasis. ■ Recommendations for social work
intervention with parents are also specified on the treatment sheet .
Having these treatment sheets available for program. monitors has been
found to be . invaluable . in providing a framework -within- which to ob-
serve the child and evaluate his progress in .the treatment program.
The structuring ,of the diagnostic staff ings in thi^way has been
immensely -helpful - in pinpointing -the needs of . a. child, .setting treatment
goals, and .outlining treatment procedures. All of these are necessary
for the effective evaluation of any program.
Periodic . Measurement ...
Only recently have - special educators .become more aware of the need
for extensive support services when dealing with -exceptional children
(Haring and Fargo,. 1969). -This is particularly .true with .emotionally
disturbed children. The. use. of program monitoring has been an integral
part of Developmental Therapy since its inception. .A. child’s needs
and behavior can. change so rapidly and in such subtle .ways • that the
therapist who is intensely involved with the child often cannot per-
ceive the changes ..quickly .enough . The feeling is that one of the primary
mistakes of traditional -treatments has been the emphasis on gross change.
Restoration of the disturbed child .comes, -in .most cases, from small bits
and pieces in the. motoric, cognitive, and emotional areas.
The periodic measurement plan at Rutland Center utilizes three di-
verse measures of -behavior. obtained unobtrusively during. the treatment
process: (1) a rating form for jthe representative objectives, (2) a
systematic observational instrument, and (3) a behavioral rating scale.
This combination -of approaches provides . a considerable amount of data;
information is obtained from three different perspectives on specified
developmental aspects of a. chi Id.
Represent at ive Objectives Rating Form . One outcome measure is
obtained from the Representative Objectives Rating Form (RORF). This
is a worksheet listing the objectives for each of the four curriculum
areas of Developmental . Therapy ; a space is provided • for a mark next to
each objective indicating .whether the objective has .been -achieved,
is currently a treatment focus, or is not yet appropriate for treatment
emphasis. In .a consensus session the educational .therapist (s) and the
monitor assess the child's progress in attaining the prescribed objec-
tives and provide the evaluation team with some quasi-objective evalua-
In addition to providing data for evaluative purposes , the com-
pletion of such a rating form yields ancillary benefits. First of all,
by recording the child ' s . progress through the representative objectives
the therapist is kept aware of his therapeutic goals and directions .
Furthermore, the task of arriving at agreement on. the form through con-
sensus provides a meaningful training opportunity for both the therapist
and the monitor.
Systematic Who-to-Whom . Analysis .. Notation * The most frequent means
of evaluating change resulting from psychotherapy is the therapist 's im-
pressions (Steisel, et a±., 1960), Such impresoions have been often
phrased in global terms, and thus specificity for adjustments in treat-
ment reprogramming have generally been difficult. Emphasis on measuring
qualitative aspects of behavior has been .properly placed on the other
two instruments . It was felt that an overt behavioral measurement ap-
proach focusing on the quantitative aspects of behavior that are subject
to observation was needed.. Such an approach, which requires a minimal
amount of subjective judgment, was chosen for its relative objectivity,
'i.e., a selected behavior occurs or does not occur.
Quantification of overt behavior is not an . innovative approach to
the measurement of behavior. This type of measurement has been defined
by Medley. and Mitzel (1963) as process,. or interaction, .analysis.
Simon and .Boyer (1970) describe a variety of observational instruments
for use with children and teachers in . classroom situations. The basic
analytical element of any observational system . is the individual inter-
acting with someone .or . something. A particular observational system
provides a method of . encoding behavior such that the resuJ.t .is meaning-
ful in the way specified by the user of the system. Many observational
systems measure primarily verbal behavior, while. few measure physical
behavior, and. fewer ■ still measure some combination of the two. Some sy-
stems require the video-taping of behavior because of the sophistica-
tion of the encoding system. A few observational systems provide for
the encoding of behavior while the behavior is occurring, such as
Spaulding’s CASES and Flanders’ and Ober’s systems (see Simon and Boyer,
The nature of the therapeutic program at Rutland Center specified
the need for an in-process encoding instrument based on. the objectives
of Developmental Therapy.. Such an instrument would enable the observer
•to concentrate on one child and his environment at any specified time.
A review of the available observational systems showed no system adap-
table to the periodic . measurement needs of Rutland Center. A who-to-
whom format was deemed necessary since an observer. needs to concentrate
his observing on one child at a time. An instrument was thus con-
structed which appears to satisfy the requirements of our situation.
This outcome measure is a behaviorally based observational in-
strument which is utilized unobti-utsivelv. The instrument, Systematic
Who-to-Whom Analysis . Notation (SWAN), is composed of twenty-six cate-
gories based on the representative objectives specified in. D.evelopmental ’
Therapy. . Each category measures some. subset of the objectives and aims
at mutual exclusiveness by encoding particular behavior in one, and only
one, category. The system as a whole also aims at exhaustiveness, al-
lowing every behavior to be .encoded into -some .category. . .
Observers are located in one-way vision observation rooms equipped
' with sound systems. The .three-second rule is employed, -^.e^. , one be-
havior is encoded .in. each three-rsecond -time period. .-Various protocol
requirements are built into the system. as described by Swan (1971).
The data are . encoded . on . a who-to-whom observation worksheet and provide
for reporting . information quickly . and .understandably
Initial .reliability .investigations -have .yielded ^rather impressive
findings. Inter-reliability coefficients (Bernstein* 1968) range from
.70 to .97. -
Clinical Qualitative Behavioral Scale . - The third instrument em-
ployed in the periodic measurement process is the Clinical Qualitative
Behavior Scale (CQBS), which is used to quantify some qualitative as-
pects of behavior. Many of the problem areas indicated in the RFCL
were translated into . objectives .measurable . in behavior terras . However,
some objectives implied -by . the RFCL (£*^* , ability . to express anger)
cannot be evaluated as simply attained, or not attained. Many such
behaviors must be viewed on a continuum. and therefore ^evaluated in
quali tative. terms. . ..It is only. whan these behaviors impair the child’s
functioning that they receive special attention. The CQBS allows for
quantification of the clinician's judgment as to the severity of. the
disordered. behavior manifested by the child.
The instrument, developed jointly by the Rutland Center psy-
chologist, and psychiatrist, is a 26- it em, ; seven-point . rating scale
anchored at . both . ends by . descriptions of degree . of impairment . In-
vestigations . of reliability are currently being- performed, and a
training program for the use of the rating scale is being developed.
Assessment and . Reprogramming .
The information obtained from measures on the three instruments'^ is
presented to those. concerned with the decision-making process regarding
the individual . child' s treatment program... These. data are summarized,
for the purpose of feedback to the staff, at different -time periods.
Data from the Sl^AN are summarized weekAy in the form of proportion of
time spent exhibiting the. various behaviors; .each child is. observed
for one minute per. week in each of four different activity periods.
Some questions which may be answered by the accumulation .of such data
week after . week . are 1) Are desired behaviors being. elicited during
each activity . period? 2). Which children are responding to which
children? 3) What activities are most stressful and/or anxiety pro-
voking? 4) Who is more dominant, the teacher or the class as a group?
and 5) Is. the activity a proper means for the child to attain his pre-
scribed objective(s)? Data .obtained from. this instrument . are also used
in a "summative" sense. The categories .are grouped so as. to reflect
"approriate" or "inappropriate" or. "neutral" overt . behaviors . Obser-
vations are recorded in the first and. last two weeks. of each ten week
period — for each child this amounts to an observation time of eight
minutes at the beginning .and eight minutes at . the end of the given time
period. .To.obtain a relatively gross picture of.change in each child’s
overt behavior, . the proportion of time spent in each of the three cate-
gory groupings is obtained this is also done by class rather than in-
dividual' child, -if - such data are requested.
The RORF is used at the middle (fifth week) and end (tenth week)
of each quarter; these forms are completed. jointly -by -the therapist(s)
and the monitor. . For. each child the number of objectives attained in
each curriculum area is obtained. This information is also examined
at a number of consecutive five-week intervals and may thus be con-
sidered, in a sense, longitudinal growth data. Data may also be sum-
marized for. each class by using the median number of attained objectives.
Data from the CQBS are collected less frequently than with the
SWAN or RORF. Consideration for each of the 26 behavioral
items is given initially at the time of intake and. again .within two
weeks after a child has begun treatment at the Center. Both completions
are for the purpose of obtaining baseline data for. planning the treatment
program. Subsequently the CQBS is completed as. a ’’post-treatment” mea-
sure to help estimate a child’s ..readiness for termination. Changes in
the ratings may be examined for each item or , after an adequate norming
sample has been observed, changes in component or factor scores (assuming
substantial reliability) may be assessed. These changes may be deter-
mined for each child or by class.
The decision-making process involves a cooperative effort on the
part of the therapist, monitor, psychologist, and evaluator. This process
may yield a new group assignment, a different emphasis in therapy, a
reassessment of the child’s environment outside of the Center, or entry
into the termination process. Hence, a recycling of the child with re-
spect to setting of treatment objectives, focus of therapy, treatment
techniques, etc., may result.
Termination and Tracking
Termination -is a process which involves a gradual decline in the
number of hours in the Center, and a gradual increase in the dependence
of the child upon -normal experiential settings to maintain appropriate
behavior. When it ..is judged that a^child should -begin .the termination
process, he is -observed several times in situ by a psychologist , a
psychiatrist > -and an educational -therapist . . -Conferences are held with
his parent(s) and, -if appropriate, with his regular c-lassroom teacher.
If deemed necessary, -additional tests (^•5,* » <^®velopmental and educa-
tional) are administered. . As . the- child’ s -eontact with Rutland Center
is gradually reduced, supportive services are encouraged -from such
agencies as Boys’ Club, recreation department, preschool and day care
It may be .possible to continue rendering service to the child
after direct Center treatment is terminated. These services make up
what is termed -"tracking.” .. The . detailed tracking -procedures are cur'-
rently being finalized with help from the social work team.
Individual tracking plans will be set up for each child.- The plans
will generally consist of a follow-up of his progress at school and
at home. This follow-up is accomplished through .a consultation service
which involves ..parent conferences , teacher conferences, .and observations
by Center staff members. Consultations are planned to occur approximately
one month, -three months , six months , .and -one year -after termination of
direct Center treatment (RFCL data from the parent(s) and school teacher
are collected at these time intervals'). Information from these con-
sultations may -indicate a need for (1) reactivating direct Center ser-
vices to the child or. parent, (2) referral services to another agency,
or (3) extended consultative help tro -the regular school teacher.
Current and Subsequent Activities
Of course, the .current evaluation plan for -the service "^to— children
component .will -be -Subjected to an ongoing evaluation itself, with modi-
fications and alternative -Strategies expected. Some of these changes
may come about as a result of the information -and -practices which will
be specified. in a near future release of -a Curriculum Guide for Develop-
mental Therapy.^ This guide will include recommendations with regard
to materials , classroom environments , types .of- verbalizations , structure
of activities , -etc. . An attempt to strengthen the evaluation process
is being made -through - numerous ongoihg investigations v For example,
a validation of the hierffl?chical order of the objectives in the RORF,
as well as norming -the objectives on selected -samples -(for indicating
age appropriateness) is currently being planned. Following such
analyses and .norming, -a.. study of the objectives as particular predictors
of emotional growth is .anticipated.
Plans for .evaluating the service-to-parents and training components
of the project are currently being formulated. Included in these plans
are instruments .measuring attitudes ■ as well as questionnaire-type in-
struments. Once.all-ofthe service programs become fairly. well defined,
and general evaluation plans corresponding to these programs have been
implemented, it will be possible to investigate relationships between
and among various curriculum, therapist, parent, and trainee variables.
1. Dr. Mary M. -Wood is acknowledged for her careful reading of an
earlier draft of this- manuscript .
2. Rutland Center is a demonstration project for the treatment of
emotionally -disturbed preschool and primary -school age children
through a. psychoeducational approach. This project is supported
in part -by . a - grant . from the’ Georgia. Department .of -Education and
by a grant. from the-U. Sv Office of Education, -Bureau of Education
■ for the Handicapped, .under the Handicapped Children Is -Early Educa-
tion Assistance Act , -P. L* -91^230 , Part C , formerly P . L. 90-538.
3. Developmental -Therapy is a psychoeducational -process for the ameliora-
tion of emotional -and -behavioral disorders in preschool children by
the simulation -of -normal childhood experiences promoting behavioral,
communicative, -Social, and cognitive developm^t. For a complete
description -see -Wood (submitted for' publication) « ■■
4. Quantification of basically subjective or qualitative aspects of
5. This list, as wall as any evaluative instruments subsequently dis-
cussed, are available from Rutland Center upon -request .
6. During intake interviews, social- workers- assist the parent in comple-
ting the RFCL, as well as obtain pertinent demographic data. Educa-
tional therapists likewise assist the regular classroom teachers in
completing the RFCL*
7. An additional source of information is -in the form of reports of
Center staff members whO' periodically visit with the individual
child’s regular or nursery school administrator and/or teacher.
and make in^the-classroom observations of the child's behavior.
8. This guide is .expected to be completed by October, 1972, and will
'• be available at that time.
Bernstein, A, L. An estimate of the accuracy (objectivity) of nominal
category coding » MOREL Monograph Series , No . . 1 , . 19 6 8 .
Cronbach,-L. J. Course improvement through evaluation ► Teachers Col-
lege . Record , .1963, 64 , 672-683.
Ebel, R. L*. Estimation of the reliability of ratings , Psychometrika ,
Haring , N . G . , and . F argo , G . A . E valuat ing -programs .for .. preparing
teachers of emotionally disturbed children. Exceptional Children ,
Kooi, Beverly Y., and Schutz, R. E. A .factor analysis.of classroom-
disturbance inter correlations , American Educational Research
Journal , -1965, 3,-37^40.
Luborsky, L. Psychotherapy. Annual Review of Psychology , 1959, 10 ,
Medley, D. M» and Mitzel, H. E. Measuring classroom. behavior by sy-
stematic observation. - In- N. L. Gage (Ed . ) Handbook of research on
teaching . Chicago: Rand McNally, 1963,. 247-328. .
Pet er son , D . R , , . and . Quay , H . iC v Behavior problem - ohecklist , - Childrens
Research Center, University of Illinois, Champaign, Illinois, 1967.
Schrupp , -M. H. , .and -Gjerde , -C . . M. Teacher growth .in attitudes toward be-
havior problems of children. Journal of Educational Psychology , 1953,
^,.2Q3r?14, ... \
Simon, A. and -Boyer , .E. G. Mirrors for behavior . Philadelphia: Research
for Better Schools, 1970.
Stake, R. E, The countenance of educational evaluation. Teachers
College Record , 1967, 68 , 523-540,
Steisel, I, M, , Weiland, I, H., Denny, J, V,, Smith, K, , Chaiken,
Nina, Measuring interaction in nonverbal psychotic children,
American Journal of Orthopsychiatry , 1960, 30 , 405-411,
Swan, W, W, The development of an observational instrument based on
the objectives of developmental therapy. Unpublished doctoral
dissertation. College of Education, University of Georgia,
Athens, Georgia, 1971,
Wood, Mary M, Developmental therapy for the preschool disturbed.
Submitted for publication, 1971,
I— J (p
I— i UJ.
^ r-^ ^
s: CO .
S 1 ?
H* O <5
. H W H
B ft H O
(B p. pi B*
B ft CO
Cli Hiffq <
M CO fH
Pi rt pi
(ft P (ft
H* P) (B
ft B P*
f+ (_i. Pb
H* (B ^
< O •«
(B ft H
(B a* <5
< p. p«
pi H (B
H* (B s:
Pl ft o
rt O Hi
p. H .
rt CO (B
(B •<: P»
pl O H
g B* 0
The Evaluation Plan for Services to Children