Skip to main content

Full text of "ERIC ED064353: An Evaluation System for a Psychoeducational Treatment Program for Emotionally Disturbed Children."

See other formats


DOCUMENT RESUME 



ED 064 353 

AUTHOR 

TITLE 



PUB DATE 
NOTE 



EDRS PRICE 
DESCRIPTORS 



TM 001 521 

Huberty, Carl J. ; And Others 

An Evaluation System for a Psychoe(5ucational 
Treatment Program for Fmotionally Disturbed 
Children. 

Aor 72 

27p. ; Paper presented at the annual meeting of the 
American Educational Research Association (Chicago, 
Illinois, April 1972) 

MF~$0.65 HC“$3.29 

^Behavioral Objectives; =<« Emotionally Disturbed 
Children; ^Evaluation Methods; ^Measurement 
Instruments; ^Psychoeducat ional Processes; Rating 
Scales 



ABSTRACT 

A general description of an overall evaluation system 
which is being implemented in a center for emotionally disturbed 
children is presented. The system is based upon three types of 
activities; planning, monitoring, and appraising. The application of 
the system to the evaluation of direct services to children is 
outlinai* The evaluation plan for the child treatment program 
involves five phases; intake, staffing, monitoring, termination, and 
tracking. Three periodic measurement instruments used during the 
monitoring process are discussed: a clinical behavioral scale 
completed by a psychologist; a behaviorally ba'^ed instrument 
completed by trained evaluators; and a rating form completed jointly 
by a monitor and therapist (s) . (Author/DB) 



TM 001 5 21 Ed 064553 



U S. DEPARTMENT Ot= HEALTH. 

EDUCATION A WELFARE 
OFFICE OF EDUCATION 
THIS DOCUMENT HAS SEEN REPRO- 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORIG 
INATING IT POINTS OF VIEW OR OPIN- 
IONS STATED DO NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU 
CATION POSITION OR POLICY 



AN EVALUATION SYSTEM FOR A PSYCHOEDUCATIONAL 
TREATMENT PROGRAM FOR EMOTIONALLY DISTURBED CHILDREN* 



Carl J. Huberty 
University of Georgia 



John P . Quirk * 

Rutland Center 
Athens, Georgia 



William W. Swan 
Rutland Center 
Athens, Georgia 



* Paper presented at AERA Annual Meeting, Division D, Chicago, April, 1972. 
t* Now at Indiana State College, Indiana, Pennsylvania 



FILMED FROM BEST AVAILABLE COPY 



ERIC 



1 



1 



All Lv.'u uat a ona 1. Syst^am for a Psychoeducational 
Treatment Program for Emotionally Disturbed Children 



ABSTRACT 



A general description of an overall evaluation system which is 
being implemented in a center for emotionally disturbed children is 
presented. The systeni is based upon three types of activities: planning, 

monitoring, and appraising. It is pointed out that in such a three- 
pronged model these activities are neither independent nor mutually ex- 
clusive; they are not only compatible, but mutually supportive. It is 
the interrelations of the three types of activities which produce the end 
product. Once the initial planning is completed, the model affords 
reassessing, modifying strategies, and reprogramming whenever desirable. 
Following decisions of reprogramming, the evaluative cycle repeats it- 
self: planning, monitoring, and appraising. It is for this purpose that 
a well developed information exchange system within the center is needed. 

Such evaluative procedures make it possible to advantageously integrate 
data collection into the decision-making process. 

The goals of the evaluation team are; 1) to assist in expressing 
questions to be answered and information to bs obtained, 2) to collect 
the necessary information, and 3) to prepare the collected information 
in a form useful for decision makers for assessing decision alternatives. 

The Information to be used in each component program is in the form of 
data that provide descriptions and judgments of anything which feeds in- 
to the progi'am (antecedents), happens during it v -cransactions) , and re- 
sults from it (outcomes), along with the contingencies among these. The 
antecedents constitute a major contribution to the planning and develop- 
ment of the evaluation strategy(ies) to be subsequently employed. It is 
a function of the evaluation team to relate the transactions to the ob- 
jectives and processes of each component. The concern with the output 
data is one of devising performance criteria, relating these data to 
the other two types of data, and formulating ‘decisions regarding worth 
and attainment of component objectives. 

The somewhat detailed application of the system to the evaluation 
of direct services to children is outlined. It is emphasized that the im- 
portant prerequisites of an evaluation svstem of a chi.M treatment nrogram arc 
that the system be easily implemented i cui usr>^u’ . 

The evaluation plan for the child treatment program involves' five 
phases: intake, staffing, monitoring, termination, and tracking. Through 

a problem check list, a langua^ common to individuals of varying back- 
grounds, from parent to psychiatrist, is used through the first two phases. 

A second language, closely allied to that of the problem check list, in the 
form of a list of treatment objectives, provides a commonality the Rutland 

Center professionals upon which a meaningful monitoring process has been 



3 



dovsloped. Three periodic moasiirement instruments used during the 
monitoring process are discussed* One instrument, a Ciinical behavioral 
scale completed by a psychologist, measures the qualitative aspects of 
behavior; a second instrument, behaviorally based and completed by 
trained evaluators, measures the quantitative aspects of behavior; and a ^ 
third instrument, a rating form jointly completed by a monitor and thern^ 
pist(s), measures both aspects of behavior. All Liiree iristrumcnts wore 
developed from the tv;o common langungn,’. mentioned previouslv. 

It was discussed how the evaluation system provides foi*’. a- periodic 
feedback of information which is useful in supporting decisions regarding 
the individual child and the treatment program. Such information, along 
with that obtained at intake, aids in deciding when the termination pro- 
cess should begin. It was discussed how further evaluation will be made 
during termination and after direct Center treatment ends. 




3 



4 



an EVALUATlOh’ SY:;TEM irOR A PSYCHOEDUCATIONAE 

1 

TREATMENT EROCRAM FOR EMOTIONA'.LY DISTURBED CHILDREN 
The Evaluation System: An Overview 

2 

A current thrust in the efforts at Rutland Center' is the develop- 
ment of an evaluation system. This system is considered an integral, 
part of the overall project rather than an adjunct to it, and the eval- 
uation personnel have taken, and are taking, an active role in the plan- 
ning, monitoring, and appraising phases oi all Center operations. Be- 
cause of this total involvement the success of the evaluation system is 
dependent upon a well developed system of information exchange which en- 
hances feedback and communication. The involvement of the evaluation 
team in the total project and its participation in the exchange of in- 
formation are depicted in Figure 1. Note that the evaluation team is 
expected to provide evaluative services (in the form of planning, moni- 
toring, and appraising) to each of four components: demonstration and 

dissemination, training, service-to-children , and service-to-parents . 
(Although data are collected for the purpose of demonstration and dis- 
semination, the following comments in this section generally pertain to 
the other three components . ) 



Insert Figure 1 about here 



The goals of the evaluation team are: 1) to assist in expressing 

questions to be answered and information to be obtained, 2) to collect 
the necessary Information, and 3) to prepare the collected information 

o 

ERIC 



4 



in d form usi-iai ior decasioi. inaxers U.r asS' :'.sing '.!eoi.sion alternatives* 

The informati' n lo used • n each component progr-am is in thQa^jform of 
data that provide d-^scrJ pt lens and judgments of anything which feeds into 
the program (anteceden- ■) , happens dur-ing it ( i rarisactions ) , and resu-its 
from it (outc:omes)j along wrth the contingencie' • among these (see Stake, 
1967). The antecedents include such inputs as trainee, child, and parent 
characteristics, referral data, environmental factors, and the psycho- 
educational curriculum and techniques. These inputs constitute a major 
contribution to the planning and development of the evaluation strategy( i- s ) 
to be subsequently employed. Involved in the transactions are the pro- 
cesses and interactions witnin and among learning or training activities, 
individuals, and materials. It is a function of the ovaluaticui team to 
relate such data to the objectives and processes of each component. The 



outputs pertain to the individual client, to the home , and co the Center. 
The concern with the output data is one of devising per-tormance criteria, 
relating these data to the other two types of data, and supporting de- 
cisions regarding attainment of component objectives, need for treatment 
modification, need for reprogramming and recycling and readiness for term! 

nation of treatment. 



The various functions and roles of the evaluation team within the 



framework of the Cenlei' are outlined in Figure 2. It is important to 

note that in the three-pronged model these three types of evaluation activities 



Insert Figure 2 about here 



ai’e neither Independent nor mutually exclusive; they are not only compatible, 
but mutually supportive. As with the three kinds of data used for the evalua- 



o 

ERIC 



f 



5 



V- 



i. 



r*- 



i7. I p . u> : ‘ v'P - / ’ ! • <- * • ■. ; ' ‘ ^ • JC * ‘ ^ 

w nr / i* •• -.'.‘ i- • 'v - • < " T ■» :i ' ’■ 



. ^ ■ rr.. . » r • t 



. r. , -r- . • i w i ■: iV^r!iv.. nfnr ^ ' vh‘ -.o.^ . r-n'' r-‘’'r' 0 '-'r.t'.it' . n - 

c/"' •.'. "iS i . I'o. ('■■?■* '..sVvP.j of re jf : .’z'. .r:":.i p'' . the ev '.vo 

■•'''r.eatc: i :o j t. * i ■J.vinriiPP’ r- >ni ’■-orin'"- , ai>( 1.'-. i-''. > r. ’■ ■ ■ •• 



eye 



nuroor.c 



l-V 



ac a v;eli deve.-opc .i u' . o '''ration exe'-a- •' 



svst'-’'’' T. f >n Pfsni 



.’.e ne 



edsd. Such ev*axU'-t vo nr’-.'ico'i.ur'S^- make : a tc. . v-.zit.a'''eou . '• 



intej^': :zte data eoilecc'on into the de'^-ision -n-'. 



, , ■ -rf-s.s . 



heeause o? theoneticaj. iimvtati ons , and JtJi’ ni'af i aip’ ethical 
reasonr.. (especially with regard to the service- l o-chJ Ldr.-en fiorp.^r-enL) , 
the evaluation plans do not caiJ. tor a con-paraz. : ve ...isczs'jment C" trear 
merits or curricula. That is. the system does nor iriciude vzhai hs found 
in. tyT'icci 'research' or ‘experimental" se-^unz>;- r.ai'eJ.v, random samples, 
constant ' treatment ‘ controlled variables, anc’ (.ornoarison or control 
groups. Rathe'*^ . the concerfz is vzith detai.Led tier. cr i ot ions and. obaer .^riop-. 
of individuals or small o nuses. The posiiion is sinv'Jar to tha^ of 

Cronhach (1963)- the ain to compare one nrograni v.iic another shouLd not 
dornin-'c e plans i cr evaiua toni evaJ.uation shuiuid >e tjr.tpari ’v conce''’nt-..d 
V 71 th the ef.toi:>'.s of thei program under sf ud'?. t ^r, j ^ind Center efiert is 
addressed to the question, ’’What changes < ao be observed in a certain .'cizvi 
.individual v/hic'n can be attributed to an Involvam.em in a cert.iji- .k.inc c' t; 
gram intervention?" Come time ago, ’ iibcrckv (j9S9 . p. 328) poir'tec- out 
It has yet to be demonstrated that control nrouos m psychotherapy re- 
search have a moi-e than very limited usefulness.’* The literature ■ -.c t.ie 
past decade has not produced much evidence to the contrai’y • 



■f. 



6 



7 



The Rutland Center evaluation methodology is not necessarily de- 
signed to yield universally valid information: the focus is on thes^ par- 

ticular treatment processes, integral parts of this psychoeducational model. 
The emphasis in the evaluation program may be likened to a current emphasis 
(controversy?) in educational measurement namely* that of criterion- 
referenced measures. Rather than comparing the performance of individuals— 
trainees, children, parents — in the Rutland program with other individuals 
(norm-referencing), criteria are being established for each individual, 
thus enabling the individual’s progi’ess to be assessed relative to himself. 
(This does not, of course, preclude the use of norm-referenced measures ob- 
tained from ’’standardized” tests to yield input data.) These criteria for 
attaining objectives are usually not determined until after the individual 
receiving services has entered the program and some assessment has been made. 
And the decision of whether or not an individual has attained a criterion 
established for him is based upon as much objective information as possible 
(test results, systematic observation, rating forms, etc.), supplemented by 
whatever clinical judgment is deemed pertinent. Such decisions are made, of 
course following discussions involving an evaluator, a teacher, a psycholo- 
gist, a monitor, and anyone else who may be familiar with the individual. 

The success of such an evaluation methodology is highly dependent upon 
explicit statements of the goals and objectives of each of the project 
components . The inputs , transactions , and outputs must dii’ectly relate 
to the general objectives of each component as well as to specific objectives 
associated with the individual trainee, child or parent. The inportance 
and role of the objectives are clearly reflected in the three-pronged 
model (planning, monitoring, appraising) discussed previously. The 
emphasis is on (measurable) objectives as guidelines for action, and on 



o 

ERIC 



f 



7 



8 



meaningful observation and description in assessing an individual’s pro- 
gress, or lack of it. 

Most of the evaluation effort extended to data has been focused on 
the treatment progx'am for the service-to-children component . Considerable 
work has been done in planning for the evaluation of the effect of the 
Developmental Therapy^program on four classes of preschool emotionally 
disturbed children at Rutland Center. The remainder of this paper dis- 
cusses the application of the evaluation system outlined above to a 
method of treatment designed to ameliorate the child’s symptoms and to pro- 
duce gains in those areas which are most debilitating to his functioning. 

Evaluation of Service to Children 

For an evaluation system to be employ®<i iri a treatment program it 
must not only be empirically sound but, more importantly , it must in 
the long run be useful for clinical practice. To be clinically useful, 
an evaluation system must be intimately tied to the philosophy and under- 
lying theory upon which the treatment program is based. This has been 
particularly difficult for traditional treatment programs which focus 
exclusively on broad hypothetical constructs related to psychodynamics. 

The emphasis at Rutland Center, however, is on problem behaviors manifested, 
or perceived, in the home and/or the school. Having a problem behavior 
orientation instead of a mental illness framework has made it possible 
to develop specific behavioral objectives for treatment planning and for 
measurement purposes. Recognizing also that qualitative aspects of be- 
havior are important, provision has been made in the evaluation system for 
the measurement of these aspects. 




8 



FILMED FROM BEST AVAILABLE COPY 



9 



In addition to the need for consonance with a theoretical base, 

.iii cvaluaiion system must be composed of procedures that can blend 
. Into the everyday, functioning of a treatment center. Any syste’- 

triai; takes an inordinate amount ref fort and does hof’fa^' 

-iitate the treatment function will soon be discarded. Practicing edu- 

caiors , psychologists, and social workers need evaluation procedures 
wiih which they can be comfortable and which help them be more effective 
in dealing with children’s problems. 

Thus, to be affective, an evaluation system must be built into the 
treatment program itself. Objective delineation of problems, setting 
treatment goals, periodic assessment of progress, and the utilization 
of objective or quasi-objective^ data for making treatment decisions 
should be not only qualities of a useful evaluation system but also 
necessary characteristics of any productive treatment program for children. 

The general goal of the servlce-to-chlldren component is; to pro- 
v'ide psychoeducational treatment experiences to referred children so as 
to enable them to better cope with their home and school environments, 
-aruirable outcome objectives’ f dr the' >childre'tt’’^nv61ve decreasing the 

•: amber and/or severity of blhd’VibrSi''-^ and improving appropriate 

■akiJl; .in curriculum areas of the psychoeducational process. 



tion system is to assist in the planning and maintenance of the treatment 
program for children. However, since the philosophy of treatment here is not 
strictly behavioristic, a potential difficulty existed at the outset. 

It was felt that the objectives must reflect both the developmental aspects 



Structuring of the Treatment Program 



The development 




ERiC 



9 



10 



of the treatment model and the qualitative aspects of behavior, and at the 
same time maintain a sufficiently behavioral orientation to allow for 
somewhat objective and reliable measurement. Extreme ..specificity in the 
st<^tement -of -objectives would have -had a limiting -effect .on the. psycho- 
educational therapists, while over-generalization would have made the ob- 
jectives difficult .to assess. 

From this demand for a balanced approach , -the list -of representative 
objectives^ resulted .. These objectives provide behavioral milestones 
around which the treatment program of a chi Id .can be -planned and monitored. 
The objectives .range - from. simple .attending and ..responding -behavior neces- 
sary for any constructive child-environment interaction. to more complex 
social skills ihch as those involving lel^dership behavior. They were de- 
veloped around the.four.^ curriculum areas pf -Developmental -Therapy: be- 

havior, communication, .socialization, .and school readiness . -An attempt 
has been made to specify .the .hierarchical order in which these behavioral 
ob j actives appear - in . the - developmental .process • This • list serves as a 
common language. useful. for . the purpose of outlining measurement -procedures 
and constructing data- collect ion instruments. This . commons 1 ty maximizes 
communication among the various staff members involved in the periodic 
measurement process. 



The Evaluation Plan 

The evaluation plan. for ..the service^to-^ohildren. component is viewed 
as consisting of . five major phases which coincide with -the flow of diag- 
nostic and therapeutic .procedures of the tipeatment progr%<' The phases 
are intake, staffing, monitoring, termination, and tracking. F.ach phase 




10 



11 

is directly supported by data collected and sumraarized by the evaluation 
team (see Figure 3). The evaluation team assists in the delineation of 



Insert Figure 3 about here 



the child’s problems during intake and staffing, provides periodic feed- 
back information necessary for . maintaining and adjusting the treatment 
program, assists in speci^ing ..termination criteria, and obtains follow- 
up information after direct Center treatment ends . 

The evaluation ..and .monitoring . effort .begins with .the - initial contact 
with parents and regular teacher and ends approximately one year after 
the child has been. terminated -from the treatment program. . .Throughout the 
diagnostic, staffing, .and -treatment . phases . of the .program. the evaluation 
system yields important .informational feedback to .the. professional staff. 
All of the professional .staff .members participate in the development of 
procedures which provide the required data. These procedures are aimed 
at increasing the. amount and usefulness of objective and quasi-objective 
data employed in making clinical judgments. 

Intake and - Staffing . . 

Many multi-disciplinary treatment teams have found it difficult to 
delineate problem areas to. the satisfaction of all involved. A common 
language , which . facilitates .communication among educators , psychologists , 
psychiatrists, social workers, measurement personnel,.parents and regular 
classroom teachers, .is essential if a .child.is to. receive maximum benefits 
of a treatment program.. Provision for such a . common language rin the deli- 

id 

ERIC 



11 



12 



neation of children’s proiilem areas is made by the Referral Form Check 
List (RFCL). 

The RFCL is a composite of behavior problems abstracted from re- 
ferral records accumulated over a two— year period. The treatment files 
ware reviewed and all referral problems for preschool and primary 
school children were listed. Over 200 behavior problems were recorded; 
from this list -many were eliminated because of . duplication of problem 
meaning. This .synthesis resulted in the check list, which is composed 
of 54 behavior problems grouped within the four curriculum areas of 
Development al. Therapy. - -A. review of. the literature (£.£. , Peterson and 
Quay, 1967; Kooi and Schutz, 1965; Schrupp and Gjerde, 1953) Indicated 

that the RFCL. contained -characteristics which .are -identical or parallel 
to those that have bean previously . investigated. • A . five-^point rating 
scale format , .ranging, from -’’High. Priority Problem’’ to ."Not a Problem 
or Not Noticed" .was selected -because such a format . (a) provides a 
range for detection .of behavioral change over .time, (b) .allows for re- 
cognition of problems .perceived .by . adults as ..’’real'.’ adjustment problems , 
and (c) permits the incorporation of clinical inference in the judgment 
process. 

Investigation of reliability of the RFCL is currently in progress . 
Inter-observer . reliability . estimates have been obtained using an intra- 
professional . group . C i.»^» , educators , psychologists , etc . ) orientation . 
Initial results are encouraging. Using the coefficient suggested by 
Ebel (1951), reliability estimates range from .46 to .76 across pro- 
fessional groups. 

During the intake procedure , ratings ^bn the RFCL are obtained from 
each staff member who is involved in the diagnostic process (educational 




12 



13 



tester, psychologist, and psychiatrist). In addition, -RFCL's are coii^ieted 
by the child^s parent(s) and regular classroom teacher. The multiple 
perception of a single, pool . of pronlem behaviors -has ..been extremely help- 
ful in facilitating the presentation ofa comprehensive picture of a 
child during. staffing. .The .evaluation. team. collects .and summarizes the 
data from. all. of the RFCL's completed. Subsequently, .at . staff ings this 
information is summarized via a RFCL profile. bar graph and summary sheet. 

The summation. of. perceived problems thus seems not only . to solidify 
thinking as a staff,. but also to reduce. the need for ■ detailed diagnostic 
reports from. each staff member. . Brief clinic staff -reports are given 
which focus mainly on. the possible etiological factors that have been de- 
rived through clinical judgment. Allowing for. multiple hypotheses in de- 
termining the source of a child’s problem has proven invaluable for main- 
taining a flexible. treatment approach. 

(Pilot tesL-.-’.ng is planned for the utilization of this same RFCL for 
the purpose of obtaining post-r treatment measures from parents and regular 
classroom teachers for the detection of problem change, or change per- 
ceived by the adults .involved. ) 

Other data are also obtained prior' to staffing. Tests measuring 
such things as social behavior, perceptual-motor development, and academic 
readiness are administered; intelligence and projective measures are also 
obtained. 

To facilitate program planning and subsequent monitoring, the staffing 
information -is recorded .on ..a three- columned treatment sheet. The first column 
contains all the high priority problems. The second column contains the 
suggested causative factors underlying the behavior problems. Tlie third 
column outlines the treatment focus with specific suggestions for be- 




13 



havioral objectives needing emphasis. ■ Recommendations for social work 
intervention with parents are also specified on the treatment sheet . 

Having these treatment sheets available for program. monitors has been 
found to be . invaluable . in providing a framework -within- which to ob- 
serve the child and evaluate his progress in .the treatment program. 

The structuring ,of the diagnostic staff ings in thi^way has been 
immensely -helpful - in pinpointing -the needs of . a. child, .setting treatment 
goals, and .outlining treatment procedures. All of these are necessary 
for the effective evaluation of any program. 

Periodic . Measurement ... 

Only recently have - special educators .become more aware of the need 
for extensive support services when dealing with -exceptional children 
(Haring and Fargo,. 1969). -This is particularly .true with .emotionally 
disturbed children. The. use. of program monitoring has been an integral 
part of Developmental Therapy since its inception. .A. child’s needs 
and behavior can. change so rapidly and in such subtle .ways • that the 
therapist who is intensely involved with the child often cannot per- 
ceive the changes ..quickly .enough . The feeling is that one of the primary 
mistakes of traditional -treatments has been the emphasis on gross change. 
Restoration of the disturbed child .comes, -in .most cases, from small bits 
and pieces in the. motoric, cognitive, and emotional areas. 

The periodic measurement plan at Rutland Center utilizes three di- 
verse measures of -behavior. obtained unobtrusively during. the treatment 
process: (1) a rating form for jthe representative objectives, (2) a 

systematic observational instrument, and (3) a behavioral rating scale. 
This combination -of approaches provides . a considerable amount of data; 
information is obtained from three different perspectives on specified 



14 



1 



developmental aspects of a. chi Id. 

Represent at ive Objectives Rating Form . One outcome measure is 
obtained from the Representative Objectives Rating Form (RORF). This 
is a worksheet listing the objectives for each of the four curriculum 
areas of Developmental . Therapy ; a space is provided • for a mark next to 
each objective indicating .whether the objective has .been -achieved, 
is currently a treatment focus, or is not yet appropriate for treatment 
emphasis. In .a consensus session the educational .therapist (s) and the 
monitor assess the child's progress in attaining the prescribed objec- 
tives and provide the evaluation team with some quasi-objective evalua- 
tive data. 

In addition to providing data for evaluative purposes , the com- 
pletion of such a rating form yields ancillary benefits. First of all, 
by recording the child ' s . progress through the representative objectives 
the therapist is kept aware of his therapeutic goals and directions . 
Furthermore, the task of arriving at agreement on. the form through con- 
sensus provides a meaningful training opportunity for both the therapist 
and the monitor. 

Systematic Who-to-Whom . Analysis .. Notation * The most frequent means 
of evaluating change resulting from psychotherapy is the therapist 's im- 
pressions (Steisel, et a±., 1960), Such impresoions have been often 
phrased in global terms, and thus specificity for adjustments in treat- 
ment reprogramming have generally been difficult. Emphasis on measuring 
qualitative aspects of behavior has been .properly placed on the other 
two instruments . It was felt that an overt behavioral measurement ap- 
proach focusing on the quantitative aspects of behavior that are subject 
to observation was needed.. Such an approach, which requires a minimal 




15 



16 



amount of subjective judgment, was chosen for its relative objectivity, 
'i.e., a selected behavior occurs or does not occur. 

Quantification of overt behavior is not an . innovative approach to 
the measurement of behavior. This type of measurement has been defined 
by Medley. and Mitzel (1963) as process,. or interaction, .analysis. 

Simon and .Boyer (1970) describe a variety of observational instruments 
for use with children and teachers in . classroom situations. The basic 
analytical element of any observational system . is the individual inter- 
acting with someone .or . something. A particular observational system 
provides a method of . encoding behavior such that the resuJ.t .is meaning- 
ful in the way specified by the user of the system. Many observational 
systems measure primarily verbal behavior, while. few measure physical 
behavior, and. fewer ■ still measure some combination of the two. Some sy- 
stems require the video-taping of behavior because of the sophistica- 
tion of the encoding system. A few observational systems provide for 
the encoding of behavior while the behavior is occurring, such as 
Spaulding’s CASES and Flanders’ and Ober’s systems (see Simon and Boyer, 
1970). 

The nature of the therapeutic program at Rutland Center specified 
the need for an in-process encoding instrument based on. the objectives 
of Developmental Therapy.. Such an instrument would enable the observer 
•to concentrate on one child and his environment at any specified time. 

A review of the available observational systems showed no system adap- 
table to the periodic . measurement needs of Rutland Center. A who-to- 
whom format was deemed necessary since an observer. needs to concentrate 
his observing on one child at a time. An instrument was thus con- 
structed which appears to satisfy the requirements of our situation. 




16 



17 



This outcome measure is a behaviorally based observational in- 
strument which is utilized unobti-utsivelv. The instrument, Systematic 
Who-to-Whom Analysis . Notation (SWAN), is composed of twenty-six cate- 

'IN 

gories based on the representative objectives specified in. D.evelopmental ’ 
Therapy. . Each category measures some. subset of the objectives and aims 
at mutual exclusiveness by encoding particular behavior in one, and only 
one, category. The system as a whole also aims at exhaustiveness, al- 
lowing every behavior to be .encoded into -some .category. . . 

Observers are located in one-way vision observation rooms equipped 
' with sound systems. The .three-second rule is employed, -^.e^. , one be- 
havior is encoded .in. each three-rsecond -time period. .-Various protocol 
requirements are built into the system. as described by Swan (1971). 

The data are . encoded . on . a who-to-whom observation worksheet and provide 

for reporting . information quickly . and .understandably 

Initial .reliability .investigations -have .yielded ^rather impressive 
findings. Inter-reliability coefficients (Bernstein* 1968) range from 
.70 to .97. - 

Clinical Qualitative Behavioral Scale . - The third instrument em- 
ployed in the periodic measurement process is the Clinical Qualitative 
Behavior Scale (CQBS), which is used to quantify some qualitative as- 
pects of behavior. Many of the problem areas indicated in the RFCL 
were translated into . objectives .measurable . in behavior terras . However, 
some objectives implied -by . the RFCL (£*^* , ability . to express anger) 
cannot be evaluated as simply attained, or not attained. Many such 
behaviors must be viewed on a continuum. and therefore ^evaluated in 
quali tative. terms. . ..It is only. whan these behaviors impair the child’s 
functioning that they receive special attention. The CQBS allows for 




17 



18 



quantification of the clinician's judgment as to the severity of. the 
disordered. behavior manifested by the child. 

The instrument, developed jointly by the Rutland Center psy- 
chologist, and psychiatrist, is a 26- it em, ; seven-point . rating scale 
anchored at . both . ends by . descriptions of degree . of impairment . In- 
vestigations . of reliability are currently being- performed, and a 
training program for the use of the rating scale is being developed. 

Assessment and . Reprogramming . 

The information obtained from measures on the three instruments'^ is 
presented to those. concerned with the decision-making process regarding 
the individual . child' s treatment program... These. data are summarized, 
for the purpose of feedback to the staff, at different -time periods. 
Data from the Sl^AN are summarized weekAy in the form of proportion of 
time spent exhibiting the. various behaviors; .each child is. observed 
for one minute per. week in each of four different activity periods. 

Some questions which may be answered by the accumulation .of such data 
week after . week . are 1) Are desired behaviors being. elicited during 
each activity . period? 2). Which children are responding to which 
children? 3) What activities are most stressful and/or anxiety pro- 
voking? 4) Who is more dominant, the teacher or the class as a group? 
and 5) Is. the activity a proper means for the child to attain his pre- 
scribed objective(s)? Data .obtained from. this instrument . are also used 
in a "summative" sense. The categories .are grouped so as. to reflect 
"approriate" or "inappropriate" or. "neutral" overt . behaviors . Obser- 
vations are recorded in the first and. last two weeks. of each ten week 
period — for each child this amounts to an observation time of eight 




18 



19 



minutes at the beginning .and eight minutes at . the end of the given time 
period. .To.obtain a relatively gross picture of.change in each child’s 
overt behavior, . the proportion of time spent in each of the three cate- 
gory groupings is obtained this is also done by class rather than in- 
dividual' child, -if - such data are requested. 

The RORF is used at the middle (fifth week) and end (tenth week) 
of each quarter; these forms are completed. jointly -by -the therapist(s) 
and the monitor. . For. each child the number of objectives attained in 
each curriculum area is obtained. This information is also examined 
at a number of consecutive five-week intervals and may thus be con- 
sidered, in a sense, longitudinal growth data. Data may also be sum- 
marized for. each class by using the median number of attained objectives. 

Data from the CQBS are collected less frequently than with the 
SWAN or RORF. Consideration for each of the 26 behavioral 
items is given initially at the time of intake and. again .within two 
weeks after a child has begun treatment at the Center. Both completions 
are for the purpose of obtaining baseline data for. planning the treatment 
program. Subsequently the CQBS is completed as. a ’’post-treatment” mea- 
sure to help estimate a child’s ..readiness for termination. Changes in 
the ratings may be examined for each item or , after an adequate norming 
sample has been observed, changes in component or factor scores (assuming 
substantial reliability) may be assessed. These changes may be deter- 
mined for each child or by class. 

The decision-making process involves a cooperative effort on the 
part of the therapist, monitor, psychologist, and evaluator. This process 
may yield a new group assignment, a different emphasis in therapy, a 
reassessment of the child’s environment outside of the Center, or entry 




It <V 



20 



into the termination process. Hence, a recycling of the child with re- 
spect to setting of treatment objectives, focus of therapy, treatment 
techniques, etc., may result. 

Termination and Tracking 

Termination -is a process which involves a gradual decline in the 
number of hours in the Center, and a gradual increase in the dependence 
of the child upon -normal experiential settings to maintain appropriate 
behavior. When it ..is judged that a^child should -begin .the termination 
process, he is -observed several times in situ by a psychologist , a 
psychiatrist > -and an educational -therapist . . -Conferences are held with 
his parent(s) and, -if appropriate, with his regular c-lassroom teacher. 

If deemed necessary, -additional tests (^•5,* » <^®velopmental and educa- 
tional) are administered. . As . the- child’ s -eontact with Rutland Center 
is gradually reduced, supportive services are encouraged -from such 
agencies as Boys’ Club, recreation department, preschool and day care 
centers . 

It may be .possible to continue rendering service to the child 
after direct Center treatment is terminated. These services make up 
what is termed -"tracking.” .. The . detailed tracking -procedures are cur'- 

rently being finalized with help from the social work team. 

Individual tracking plans will be set up for each child.- The plans 
will generally consist of a follow-up of his progress at school and 
at home. This follow-up is accomplished through .a consultation service 
which involves ..parent conferences , teacher conferences, .and observations 
by Center staff members. Consultations are planned to occur approximately 
one month, -three months , six months , .and -one year -after termination of 
direct Center treatment (RFCL data from the parent(s) and school teacher 




19 



21 



are collected at these time intervals'). Information from these con- 
sultations may -indicate a need for (1) reactivating direct Center ser- 
vices to the child or. parent, (2) referral services to another agency, 
or (3) extended consultative help tro -the regular school teacher. 



Current and Subsequent Activities 

Of course, the .current evaluation plan for -the service "^to— children 
component .will -be -Subjected to an ongoing evaluation itself, with modi- 
fications and alternative -Strategies expected. Some of these changes 
may come about as a result of the information -and -practices which will 
be specified. in a near future release of -a Curriculum Guide for Develop- 
mental Therapy.^ This guide will include recommendations with regard 
to materials , classroom environments , types .of- verbalizations , structure 
of activities , -etc. . An attempt to strengthen the evaluation process 
is being made -through - numerous ongoihg investigations v For example, 
a validation of the hierffl?chical order of the objectives in the RORF, 
as well as norming -the objectives on selected -samples -(for indicating 
age appropriateness) is currently being planned. Following such 
analyses and .norming, -a.. study of the objectives as particular predictors 
of emotional growth is .anticipated. 

Plans for .evaluating the service-to-parents and training components 
of the project are currently being formulated. Included in these plans 
are instruments .measuring attitudes ■ as well as questionnaire-type in- 
struments. Once.all-ofthe service programs become fairly. well defined, 
and general evaluation plans corresponding to these programs have been 
implemented, it will be possible to investigate relationships between 
and among various curriculum, therapist, parent, and trainee variables. 

o 

ERIC 



20 



22 



FOOTNOTES 

1. Dr. Mary M. -Wood is acknowledged for her careful reading of an 
earlier draft of this- manuscript . 

2. Rutland Center is a demonstration project for the treatment of 
emotionally -disturbed preschool and primary -school age children 
through a. psychoeducational approach. This project is supported 
in part -by . a - grant . from the’ Georgia. Department .of -Education and 

by a grant. from the-U. Sv Office of Education, -Bureau of Education 
■ for the Handicapped, .under the Handicapped Children Is -Early Educa- 
tion Assistance Act , -P. L* -91^230 , Part C , formerly P . L. 90-538. 

3. Developmental -Therapy is a psychoeducational -process for the ameliora- 
tion of emotional -and -behavioral disorders in preschool children by 
the simulation -of -normal childhood experiences promoting behavioral, 
communicative, -Social, and cognitive developm^t. For a complete 
description -see -Wood (submitted for' publication) « ■■ 

4. Quantification of basically subjective or qualitative aspects of 

behavior. -. 

5. This list, as wall as any evaluative instruments subsequently dis- 
cussed, are available from Rutland Center upon -request . 

6. During intake interviews, social- workers- assist the parent in comple- 
ting the RFCL, as well as obtain pertinent demographic data. Educa- 
tional therapists likewise assist the regular classroom teachers in 
completing the RFCL* 

7. An additional source of information is -in the form of reports of 
Center staff members whO' periodically visit with the individual 
child’s regular or nursery school administrator and/or teacher. 




21 



% 



23 



and make in^the-classroom observations of the child's behavior. 

8. This guide is .expected to be completed by October, 1972, and will 
'• be available at that time. 



o 

ERIC 



22 



24 



REFERENCES 

I 

Bernstein, A, L. An estimate of the accuracy (objectivity) of nominal 
category coding » MOREL Monograph Series , No . . 1 , . 19 6 8 . 

Cronbach,-L. J. Course improvement through evaluation ► Teachers Col- 
lege . Record , .1963, 64 , 672-683. 

Ebel, R. L*. Estimation of the reliability of ratings , Psychometrika , 
1951,-^,-407^424. 

Haring , N . G . , and . F argo , G . A . E valuat ing -programs .for .. preparing 

teachers of emotionally disturbed children. Exceptional Children , 
1969, .157-169. 

Kooi, Beverly Y., and Schutz, R. E. A .factor analysis.of classroom- 
disturbance inter correlations , American Educational Research 
Journal , -1965, 3,-37^40. 

Luborsky, L. Psychotherapy. Annual Review of Psychology , 1959, 10 , 
317-344. . 

Medley, D. M» and Mitzel, H. E. Measuring classroom. behavior by sy- 
stematic observation. - In- N. L. Gage (Ed . ) Handbook of research on 
teaching . Chicago: Rand McNally, 1963,. 247-328. . 

Pet er son , D . R , , . and . Quay , H . iC v Behavior problem - ohecklist , - Childrens 
Research Center, University of Illinois, Champaign, Illinois, 1967. 

Schrupp , -M. H. , .and -Gjerde , -C . . M. Teacher growth .in attitudes toward be- 
havior problems of children. Journal of Educational Psychology , 1953, 
^,.2Q3r?14, ... \ 

Simon, A. and -Boyer , .E. G. Mirrors for behavior . Philadelphia: Research 

for Better Schools, 1970. 




23 



25 



Stake, R. E, The countenance of educational evaluation. Teachers 

College Record , 1967, 68 , 523-540, 

Steisel, I, M, , Weiland, I, H., Denny, J, V,, Smith, K, , Chaiken, 
Nina, Measuring interaction in nonverbal psychotic children, 
American Journal of Orthopsychiatry , 1960, 30 , 405-411, 

Swan, W, W, The development of an observational instrument based on 
the objectives of developmental therapy. Unpublished doctoral 
dissertation. College of Education, University of Georgia, 
Athens, Georgia, 1971, 

Wood, Mary M, Developmental therapy for the preschool disturbed. 
Submitted for publication, 1971, 




24 






c 

•o 

o 

D 

H* 






.. ^ 

rf 

fi) ^ 



cr w 
fl) o 



H 

H* 



04 P- 



I 

W 

CO 

rt 

1 

2 

•X5 



(^ 

W 

(D 

D 

r+ 

D- 

O 

CO 



o 

a 

rt 

a> 



o> 






■S & 



rt 

& 



0) 

O 

Q 



(O 

3 

rt 

(0 



w 

8 

0> 

QJ 

rt 

O 

3 



§ 

H* 

O 
(U 
rt 

H* 

O 

2 



H* 

3 O 
rt 
CO 



rt 

?* 

fU 

f+ 



T3 

§ 

a 



S' 



H* 

O 

rt» 

CO 



H* 

a 

__ H* 

c o 

H. p> 
**3 rt 



s, 



O 

o 



o 

Q 



H* 

O 



P 



rt 

3T 

P 



pi 
Pi P 



rt 

H* 

O 

3 

H 

H* 

3 

C/) 

rt 

P) 

rt 



CO 

rt 

9 

O 

H) 

M 



^ rij 

O 



H. 



rt (^ 

H. 

O H 
3 

a 

(d 



26 






> 

<x) 

111 

H* 

CO 



s 

0 

o 

H* 

H- 

3 

(W 



•D 

l~i 

tu 

§ 

H* 

(g 



> H 

2.S 

rr tP 
H* (l> 

H* O 
rt- t-h 
»< 



CO 






CS 


H 3 


P 


ft 


P 


p ns 


H 


H 


< 


0 ^ 


P 


P 


H- 


'C o 


rt 


P 


CO 


O H 
H ft 


(ft 


3 


P 


H* 


C 


(ft 


G 


3 Pi 


3 


3 




Oq (ft 


rt 


rt 


H* 



ERIC 



G 

tn 

rf- O 

C 

(/) 

w 

0) H* 

»o o 

H* D 
to 

rt' s: 

p. 

rt 

P- 

H* 

(b 

O 

rt 



•t:? 

CO 

i< 

o 

p- 

o 

H 

O 

GO 

H- 

CO 

ft 



§ 

a 



0 

1 

CD 

(0 

ft 

o 

o 

tr 

i-j* 

o 

ft 

(t> 



& 

0 ) 

H 

CO 

H- 

CO 

o 

*-h 

O 

0 

H 

H 

O 

rt 

0> 

P- 

P- 

0) 

rt 

0) 



rt 
O O 
tr *:! 

CJ« 

O 0) 
O 

ft Hi 

5'§ 

<9 

CO O 
O 

CJ* 3 
W O 
<! H 
o c: 

cr H* 



M > 
3 CO 
*d CO 

I— J (p 

(9 (» 
3 CO 

3.'S 

r+ T 
(U O 

f+ •« 

H« O 
O CO 
S (D 
H 

<5 0 

pi CT* 
I— i UJ. 

g o 
O 

r+ r+ 



O 

3 



CO 






P Pi 


CO 


(B B 
(B OP 


i ft 




B _ 


p. 0 




rt 


3 




Pl S' 


P O 




rt pi 


ft 3 




ft ft 


H« ft 




Pl 


0 O 




t:* 


3 Q 




B 


3 




© 


P 




Ol< 


CO 

«# 


S-i 


%-v- 



^ r-^ ^ 



^ » 
s: CO . 

H.*d 
f+ fi> 

H- 

§ 



CO 

ft 

PJ 

Hi 

Hi 



O 

Hi 

O 

O 

H 

H* 

0 
ft 

et* 

p- 

fu 

5) 

1 

pi 

CO 

o 

p 

CO 

CO 

H- 

§ 



g 

g 




e, 






ft ft 
^ H* 

s-g 

»d 0) 

:*I5- 

CO 



o 

o 

H 
(» H 
Hi (D 
Hi O 
(ft ft 
O 

ft pu» 

BL 

ft 

ns p 






H* (B 
O 4 

O. <4 

H« 0) 
O H« 
S 

PJ H« 
CO 0 
CO oq 

CO r+ 

a o 

S 1 ? 

r+ (6 
w p) 
rt 

% 

{3 

rt 






O EH 

0 Cli 

H CD 
H O 
(B r+ 
O H« 
f+ Hi 

ty* 

pi H« 
CO 0 
(B Cl' 
H H* 
H« < 
B H* 
(B &• 

a s 

pj H 
rt 

P) ^ 



Pl 

<B 

B 

ft 

O 

B* 

i-J« 

(ft 

o 

ft 

H- 

c 

(ft 

«co 



CO 

O 

g u 

CO P) 

p. p. 

O B 
B (B 

— ® 
s: CO 

H* 

ft 

B* 

3 

s 

H* 

S 



(B 

CO 

ft 

H« 

O 

B 

g 



P 


o 


Hi 


< 


0 


P 


H 


2 


H 


H 


H 


3 


(B 


»d 


P 


O 


rt 


rt 




O 


(B 


H 


H 


pi 






Pu 


3 

rt 


pl 


CO 


IP 


rt 


Si 




pl 


p 




3 


• CO 




Pi 



H* O <5 
. H W H 

p* (U 

B ft H O 
(B p. pi B* 
B ft CO 
gOq g.(B 

Cli Hiffq < 
O P> 
Hi ft 

6 t!‘ 
o 

B B 



^ I 

pi 
H 
(B 
CO 



26 





CO 

ft 

Ei, 

Hi 

O' 

H« 

CO 

I 



2.^ 

rt 

(ft H 
H (ft 



d 

P 

0 

CO 



3 

(ft 

3 

ft 

H* 

O 

d 



c Pi 
3 (ft 

(ft CO 
d H- 
rt (ig 
P o 

rt 

H- 

O 

o 

? 

? 

PJ 

H» 

£ 

ft 

H* 

§ 



M CO fH 
Pi rt pi 
(ft P (ft 
3 rt 
rt (ft 
H* 

^ij 



!§ B- 

CO < 
O 

p.g. 
IP 



s 



(ft 

p 



3 

rt 

H* 

Hi 



P* 



§ 

ft 



Pi 



P 

(0 

CO 

p 

CO 

p 



g 



pi 
CO 
CO 

(B CP 
CO O 
CO pi 
H 
CO 
PJ 

H* P) (B 
ft B P* 
(B CO 

i S-g 

f+ (_i. Pb 

H* (B ^ 
< O •« 
(B ft H 

Bwi H* 

CO (B 
ft CO 



8- 



pi 

rt 

JS 

(W 

H- 

P 

.CO 



CO 



» fd'* 

(B 

(B a* <5 

< p. p« 
pi H (B 

H* (B s: 
B O 
Pl ft o 

rt O Hi 
p. H . 
O 



o 

H* 

Q 

p d 
l-» CO 
3 (0 
p p. 
H* O 
H- 3 
Q ^ 



O 

Oq 

H* 

CO 

rt 

s 

eii 



g 

o 

ft 

H* 

3 

O 

H 

P 



B (B 


B 5 


»B CO 




rt CO (B 


rt rt 


(B •<: P» 


P B* 


pl O H 


fii ^ 


g B* 0 


g Pi 


0 B* 


t;* 




H *»• 


H 


ts 


0 


P 


p 


OP Pi 


0 


ft 


H« H* 


ft 


S' 


CO CO 


p 


Q 


rt O 


H 


pi 


«* P 


V 


CO 




rt M 


ns 


<B H« 


p 


Pl 0 


*<5 


O B 


o 


B* 


tr 



HI 

0^ 

W 

trj 

> 

f-* 

a 

> 

o 

z: 

!?d 

O 

o 

td 

CO 

CO 



f- 

r-; 



Figure 3 

The Evaluation Plan for Services to Children 





27