Skip to main content

Full text of "Evaluation of student learning outcomes due to self-guided engineering analysis of surroundings"

See other formats


121st ASEE Annual Conference & Exposition 


_of _ 

Engineering 


Indianapolis, IN 
June 15-18, 2014 


*»»»»»»»»»»»»»»} 


Paper ID #8679 


Evaluation of Student Learning Outcomes Due to Self-Guided Engineering 
Analysis of Surroundings 

Dr. Devin R. Berg, University of Wisconsin - Stout 

Devin Berg is an Assistant Professor and Program Director of Manufacturing Engineering in the Engi¬ 
neering and Technology Department at the University of Wisconsin - Stout. 


©American Society for Engineering Education, 2014 



Evaluation of Student Learning Outcomes Due to Self-Guided 
Engineering Analysis of Surroundings 


Abstract 

Inquiry-based learning was explored in an introductory mechanics course through the assignment 
of weekly tasks where students analyzed their surroundings within the context of the course 
curriculum. The deliverables for each assignment consisted of a photograph or video and a textual 
description. Submissions were collected first using course management software and later on the 
micro-blogging platform Twitter. Performance for each student was assessed using both course 
examinations and a concept inventory quiz. Based on these results it was found that students in 
the treatment group did not perform significantly better than students in the control group. 
However, the use of Twitter in the course was found to have benefits in terms of real-time student 
interaction and immediate instructor feedback. 


Introduction 

Inquiry-based learning is an educational approach that allows the student to take ownership over 
the education process by self-identifying a problem and formulating their own solution 1,2 . The 
application of this method of teaching was explored in an introductory mechanics course taken by 
both engineering and engineering technology students. 

Students were tasked with applying the principles of fundamental engineering analysis to objects 
found in their normal surroundings over the course of the semester. By asking students to 
complete assignments where they had to apply engineering analysis to an everyday object, I 
intended for the students to look beyond their textbook and relate the course material to their 
surroundings. Further, I hoped that students would begin to view their study of the engineering 
curriculum as a continuum rather then as a series of discrete pieces which begin and end with the 
assignment and submission of homework sets. Similar work by others has demonstrated success 
in getting students to make the connection between the classroom and the “real world” 3 . 

A preliminary study was conducted using this concept for a single assignment involving static 
equilibrium in the same course 4 . Through this effort it was revealed that, in general, students 
enjoyed completing the assignment and the ensuing class discussion was more fruitful than with 
other course topics. As a result, the concept was adopted more fully into the course with multiple 
assignments throughout the semester. The results of this expanded study are presented here. 



The deliverables for these assignments consisted of either a photograph or video of an object or 
situation that demonstrated the concepts relevant to the week’s course material accompanied by a 
brief description of what was depicted in the photograph/video. Examples of students’ work will 
be presented along with discussion of lessons learned and recommendations for the use of this 
method in the future. Evaluation of student learning outcomes was conducted through the 
issuance of pre- and post-assessments using the Concept Assessment Tool for Statics (CATS) 5 as 
well as performance on course examinations. Comparisons will be made between a treatment 
group, which was subject to the analysis assignments, and a control group, which did not 
complete the analysis assignments. 


Methods 

The application of this method of teaching was explored in an introductory mechanics course 
taken by students from both an engineering program and an engineering technology program. As 
this course is generally taken early in a student’s undergraduate program, they often experience 
difficulty grasping the concepts presented and connecting them with real world experiences. To 
help promote a deeper understanding of these concepts, students were tasked with taking a look at 
their surroundings while considering material presented in lecture. The specific directions 
provided are given in Textbox 1. Students were divided into two groups corresponding to 
registration in equivalent sections of the same course. One group served as a control group and 
the second group served as the treatment group. Both groups were subject to identical curriculum 
and assessment with the addition of the assignments described here given to the treatment 
group. 


Submit one original post per week (photo/video + text) giving an example of something that 
demonstrates the concepts discussed in that week’s classes. 

Textbox 1: Assignment instructions as provided to the students. 

Collection of student submissions took two forms. During the first part of the semester (first three 
submissions), student submissions were collected as uploads to a folder on the course’s learning 
management system (LMS). During the latter portion of the semester (final four submissions), 
Twitter (http://www.twitter.com) was utilized as a means to both collect and promote discussions 
around student submissions. On Twitter, students were asked to include a hashtag (#mech2 93) 
with each of their posts (tweets) to provide a means of quickly sorting and organizing relevant 
posts. The transition from collecting submissions on the LMS to Twitter was made in order to 
foster easier out-of-class discussion and communication surrounding a given student submission. 
Similar work has suggested that the use of Twitter helps facilitate student engagement outside of 
class and potentially improves course performance 6,7 . While the literature on the use of of Twitter 
in the classroom is emerging, recent studies have found the platform functional for promoting 
concise expression of ideas, critical reading and writing skills, stronger student-teacher 
relationships, self-learning in an informal environment, and accountability among other benefits. 
Conversely, using Twitter in the classroom has potential disadvantages such as distracting 
content, overly constraining character limitations, and privacy concerns 8 . Each of these items 




must be considered when assessing the use of Twitter in the classroom and how the integration of 
such a tool into the course curriculum might affect student performance. 

For both collection methods, students were asked to produce one original submission on an 
approximately per week schedule corresponding with the submission deadlines for their normal 
homework assignments. Each original submission was expected to include a photograph or video 
and a brief descriptive statement that demonstrated the concepts discussed in that week’s lectures. 
After the transition to Twitter posts, students were also asked to submit at least two comments on 
the posts of their classmates. 

To facilitate archiving of student Twitter posts related to the class, all posts containing the 
#mech2 93 hashtag were collected and analyzed using the Twitter Archiving Google Spreadsheet 
(TAGS) 9 . This tool allowed for automated collection of all tweets tagged appropriately along 
with the corresponding time stamp and performed high level analysis of the connections 
(mentions) between tweets. 

Students in both the control and treatment groups were given both pre- and post-assessments 
using the Concept Assessment Tool for Statics (CATS) 5 . A comparison of these assessments 
allows for a quantification of any possible variation between groups in terms of concept retention. 
The CATS evaluates knowledge in nine areas: 

1. Interactions at roller connections. 

2. Forces on systems of bodies. 

3. Interactions at pin and slot connections. 

4. Newton’s 3rd law. 

5. Representation of loads at standard connections. 

6. Coulomb’s law of friction. 

7. Static equivalence between forces, couples, and combinations. 

8. Interactions between friction-less bodies in contact. 

9. Static equilibrium. 

In addition to the CATS, performance on course examinations for each group was recorded as a 
group mean and standard deviation. 


Results 

On the first three assignments, submitted using the LMS dropbox, a total of 79 student 
submissions were collected. Consistent with what was found previously 4 , the submissions for this 
task gathered from the students exhibited a wide range in terms of quality and imagination. One 
example of a student submission is shown Fig. 1. Here it is shown that annotation (as well as 
some creativity) was used (Fig. lb) to highlight key features of the bridge truss shown and aid in 
relating the photograph to course material. 




Figure 1: Student submitted photographs of a bridge truss both (a) without and (b) with annotation. 























Other student submissions, such as shown in Fig. 2, included self-constructed free body diagrams 
to help explain the subject of the submitted photograph. The inclusion of this additional 
information suggested that the student was thinking critically about the situation depicted in their 
photograph and attempted to model the situation as seen in lecture. 

As per the assignment instructions, students were also asked to submit a textual description of 
what they were submitting each week. Similar to the submitted photograph itself, the textual 
descriptions also varied in thoughtfulness and completeness. An example of a student submitted 
paragraph is shown in Textbox 2 which accompanied the photograph shown in Fig. 3. In this 
example, the student constructed a three-dimensional system in equilibrium using a water bottle 
and the cord from a pair of headphones. Submissions of this type demonstrated student initiative 
to think critically about the course material and then manufacture a situation to fit within the 
context of the lecture content for a given week. 


For my three dimensional equilibrium photo, I chose to tie an end of my headphones to a hanging 
bottle. The headphones were then secured to a cabinet door that created a tension on the bottle. 
By the use of construction lines, we can see that the tension on the bottle exists in the direction of 
<x,-y, z>, creating a moment at the point from which the bottle hangs <0, 0, 0>. 

Textbox 2: Student submitted description to accompany Fig. 3. 

For the final four assignments, the method of collecting student submissions was transitioned to 
the micro-blogging platform Twitter. During this period, 363 tweets were collected in the archive 
over the course of the four assigned Twitter postings with 239 (66%) of them being replies 
(comments). It is estimated that approximately 80% of the total tweets submitted for the class 
were captured by the archive. Those that were missed were due to the student omitting the 
hashtag from their tweet thus making it invisible to the archiving script. It was observed that the 
majority of the tweets that were not collected in the archive were comments rather than original 
posts. A plot of the the tweet volume versus time is shown in Fig. 4. The four peaks in tweet 
volume correspond approximately with the due dates for each of the four assignments. It can also 
be seen that between submission deadlines, the tweet volume, while reduced, did not drop to zero. 
Indicating that discussions were occurring on the Twitter platform throughout the week. This has 
the added benefit of insuring that the course material is constantly being put in front of the 
students thus increasing their exposure time to the curriculum. 

An example of a student post submitted via Twitter is shown in Fig. 5. One characteristic of the 
average student Twitter submission was that the postings were typically simpler than was 
demonstrated for the LMS dropbox submissions. This was likely due to the length limitations for 
tweets (140 characters) as well as the ease of mobile posting. However, the ability to post from a 
smart phone is seen as a strength of this collection method because it helps the student to easily 
capture a moment, post it publicly, and receive immediate feedback from his or her peers. Further, 
the use of Twitter as a platform made it easier to post other forms of media such as video as 
exemplified in Karcheski 10 , Karcheski 11 , Karcheski 12 , and Hertz 13 . The student posts referenced 
here made use of Vine (a short-form video sharing site) and YouTube to demonstrate course 
content from the student’s surroundings. 

Comparing the submissions collected during the first three assignments using the LMS dropbox 






|| Weight of Projector (Lb) 


(b) 

Figure 2: Student submitted photograph of (a) an overhead projector support and (b) a free body 
diagram representation of the support. 
















Figure 3: Example of student submitted photo of a constructed system in equilibrium. 



Figure 4: Volume of tweets over time. 



































Figure 5: Example of student submitted photo posted on Twitter 14 . 







with those collected on Twitter it is evident that the dropbox submissions often contained more 
thoughtful submissions as shown in Figs. 1 and 2. However, it was the experience of the 
instructor that this method of submission made it difficult to have meaningful discussions around 
the student submissions due to the elapsed time between students completing the assignment and 
the following class session. Comparatively, using Twitter as a platform for student submissions 
and any subsequent discussion resulted in submissions that contained less detailed analysis (Fig. 
5). However, the Twitter platform permitted the conversation to happen in real-time while the 
material was still relevant. An example of this can be seen in Trueblood 15 where an initial student 
post drew follow up discussion from both the instructor and other students. 


Assessment 

Quantitative data was collected from both the treatment and control groups. The results gathered 
from both initial assessment and subsequent performance on course exams is presented in Table 
1. Data is presented as average group performance for each group along with standard deviation 
for each average score. The results show that students in each group performed at a similar level 
for the initial assessment. It is apparent that for each subsequent performance evaluation, the 
treatment group performed at a higher level than the control group. However, the variations in 
performance between each group were not statistically significant based on the standard 
deviation. This indicates a correlation between participation in the assigned tasks and course 
performance but with a need for further study to improve confidence. 

Table 1: Quantitative performance assessment for both treatment and control groups. 


Section 

Pre-CATS 

Exam 1 

Exam 2 

Exam 3 

Post-CATS 

Treatment (N=36) 
Control (N=24) 

22.5 ± 9.7 

21.5 ±7.0 

79.3 ± 10.3 
70.9 ± 12.3 

76.9 ± 9.6 
71.5 ±7.8 

67.9 ± 11.2 
60.1 ±9.2 

33.6 ± 11.3 
31.3 ± 15.4 


Using the Concept Assessment Tool for Statics (CATS), students within the treatment (N=36) and 
control (N=24) groups were evaluated both prior to treatment and after treatment. A direct 
comparison of pre- and post-CATS scores for each group using the group mean is shown in Fig. 

6. It is seen that the net gain for each group on the basis of CATS scores alone demonstrates little 
difference. The treatment group had a 49.7% performance gain while the control group had a 
45.6% performance gain measured as percentage of pre-CATS score. 

Breaking down the pre- and post-CATS scores by student and plotting provides the same result as 
shown in Fig. 7. The solid line in each plot represents the plot locations where pre- and 
post-CATS score were equivalent for a given student. Data points above the line indicate an 
improved score with greater distance from the line correlating to greater improvement of score. 

As can be seen, there is little measurable difference between the treatment and control groups. 
Further, for both groups there was little variation in gains between lower performing students and 
higher performing students as indicated by the relative uniformity of score increases across the 
range of pre-CATS scores. 




□ Pre-test □ Post-test 


Figure 6: Comparison of pre- and post-assessment using the CATS. 


By looking at student performance broken down by each of the nine categories evaluated by the 
CATS, it was found that the greatest performance gains were made in the categories of 
“Representation of loads at standard connections.” and “Interactions between friction-less bodies 
in contact.” This was found to be true for both the treatment group, with a 202.6% and a 124.7% 
increase respectively, and the control group, with a 157.2% and a 131.7% increase respectively, 
measured as percentages of the pre-CATS score. This result is likely due to the prevalence of 
these two categories throughout the course curriculum. 


Discussion 

The use of an inquiry-based learning task in an introductory mechanics course was found to be 
helpful for both the students and the instructor. Feedback gathered from the students suggested 
that despite some initial skepticism, they found participation in these assignments to be enjoyable 
and a welcome deviation from the normal homework routine. There was no clear preference 
between the LMS dropbox style of submission collection and the use of the Twitter platform. 
From the instructor’s perspective, the use of Twitter has several advantages including automated 
collection, distribution, and statistical analysis. 

The use of this style of assignment made it possible to follow along with the classes progress 
during the semester and attempt to identify misunderstandings early and address them in lecture 
immediately. For example, explaining the meaning of an internal moment as related to the student 
submission in Fig. 3. Further, the use of Twitter allowed the instructor to answer questions 
outside of class more efficiently or ideally allow other students to answer their classmates’ 
questions. By sorting through the history of Twitter posts 16 , it was found that students also used 






































Post-test Score Post-test Score 



(a) 



(b) 

Figure 7: Pre- and post-CATS results for both (a) the treatment group and (b) the control group. 





























the course hashtag to share links to online resources while preparing for exams as well as to post 
YouTube videos as evidence to support their argument for best car make on the basis of quarter 
mile time. As the instructor, I occasionally used the hashtag to share interesting links relevant to 
the course curriculum to provide students with further reading. 

Despite the lack of any clear evidence indicating improved learning outcomes, the use of this 
form of student assignment was a positive experience for both the students and the instructor. On 
several occasions students would approach me to tell me about something they saw that they 
thought was interesting, even when on vacation as shown in the example in Fig. 8. As previously 
mentioned, when comparing the LMS dropbox submissions with the Titter posts, it was found 
that the Twitter posts in general contained more simplistic analysis of a given situation. However, 
the ensuing discussion surrounding a post was often more meaningful and timely than was 
possible with the LMS submissions. 

As a result of this experience, it is planned to continue the use of this form of journaling 
assignment in future offerings of this course as well as in other similar courses. The submission 
method used for these assignments will continue to be the Twitter platform for benefits of 
real-time communication and automation of the collection and distribution process. One of the 
future goals for this work is to use these assignments to encourage students to engage with the 
online engineering community through communication with practicing engineers that use Twitter. 
This will be done by sharing an instructor curated list of engineers with active Twitter accounts. 
Further, since improved student engagement with the course curriculum is expected 17 but not 
effectively measured through exams or with the CATS, future efforts in this area will attempt to 
quantify student engagement through participation in these activities such as with a self-efficacy 
survey. 



Figure 8: Student submission collected while traveling. 






Acknowledgements 


The author would like to thank the UW-Stout students in the Fall 2013 semester sections of 
MECH-290/293 for their photographic and textual contributions. 


References 

[1] L Magnussen, D Ishida, and J Itano. The impact of the use of inquiry-based learning as a teaching methodology 
on the development of critical thinking. The Journal of Nursing Education, 39(8):360—364, November 2000. 
PMID: 11103974. 

[2] Michael Prince. Does active learning work? a review of the research. Journal of Engineering Education, 93(3): 
223-231,2004. doi: 10.1002/j.2168-9830.2004.tb00809.x. 

[3] Johanna J. Hopp. Journal writing in the physics classroom. The Physics Teacher, 46(7), 2008. doi: 
10.1119/1.2981300. 

[4] Devin R. Berg. Experiences with inquiry-based learning in an introductory mechanics course. In 2013 ASEE 
North Midwest Section Conference , pages 318-324, Fargo, ND, October 2013. ASEE. 

[5] Paul Steif. Concept assessment tool for statics (CATS), 2010. URL https : //cihub . org/resources/3. 

[6] R. Junco, G. Heiberger, and E. Loken. The effect of twitter on college student engagement and grades. Journal 
of Computer Assisted Learning, 27(2): 119-132, 2011. doi: 10.1111/j.l365-2729.2010.00387.x. 

[7] Ilkyu Ha, Jason J. Jung, and Chonggun Kim. Influence of twitter activity on college classes. In Costin Badica, 
Ngoc Thanh Nguyen, and Marius Brezovan, editors, Computational Collective Intelligence. Technologies and 
Applications , volume 8083 of Lecture Notes in Computer Science , pages 612-621. Springer Berlin Heidelberg, 
January 2013. doi: 10.1007/978-3-642-40495-5_61. 

[8] Amandeep Dhir, Khalid Buragga, and Abeer A. Boreqqah. Tweeters on campus: Twitter a learning tool in 
classroom? Journal of Universal Computer Science, 19(5):672-691, 2013. doi: 10.3217/jucs-019-05-0672. 

[9] Martin Hawksey. Twitter archiving google spreadsheet TAGS v5, February 2013. URL 

http://mashe.hawksey.info/2013/02/twitter-archive-tagsv5/. 

[10] Tyler Karcheski. #mech293 like the ball toss problem, water is shot from a height with a initial velocity, gravity 
then acce... https://vine.co/v/hUjFiZiHZvl, November 2013. URL 

https://twitter.com/tylerkarch%5f/status/405399899883450368. 

[11] Tyler Karcheski. #mech293 the door assist is similar to a 4-link arm, when the electric motor opens the door. 
https://vine.co/v/hQUOvpI7xM5, December 2013. URL 

https://twitter.com/tylerkarch%5f/status/4104371394 62381568. 

[12] Tyler Karcheski. The fan has a lower velocity when it’s turned on due to inertia and friction. #mech293 
#acceleration https://vine.co/v/hTgr6zpuu3j, November 2013. URL 

https://twitter.com/tylerkarch%5f/status/400273307528093696. 

[13] Dillon Hertz. Noticed that MineCraft simulates gravity rather well http://youtu.be/eqabldwV9Es #mech293, 
November 2013. URL https : / /twitter . com/DillonHertz/status/4 0037 432 8765591552. 



[14] Tyler Webb. Saw this as i was walking back from classes, you could analyze the velocity at the top as the arm is 
lowered.#mech293 pic.twitter.com/xGdgldiOxc, December 2013. URL 

https://twitter.com/webbt0498/status/410563673304743936. 

[15] Olivia Trueblood. Solving the internal forces in a dome would be tricky! the roman’s figured it out though 
#mech293 pic.twitter.com/vTmoEiGzw3, October 2013. URL 

https://twitter.com/xtravgnoliveoil/statuses/395276149569253378. 

[16] Twitter / search - #mech293. URL https : //twitter . com/search?q=%23mech2 93&f=realtime. 

[17] Laura McCullough. Science news and the science classroom. Journal of College Science Teaching , 36(3): 
30-33, January 2006. ISSN 0047-23IX.