Meta-Evaluation of a Case Study Evaluation
For my second artifact, I chose the group assignment from MDDE 617, Program Evaluation. This assignment was completed with a partner and involved doing a critical review of a fictitious evaluation report regarding a new online educational program. The program evaluation was provided by our instructor as a case study and was based on John Owen’s evaluation structure, which involves five categories, or Forms. The critical review was in the form of a paper that evaluated the case study based on which of Owen’s Forms it had used, descriptive elements of its developmental stage, and analysis of how well the Form was applied, and whether that application was sound given the program context. I chose this assignment for my portfolio because it was one of the areas in the program where my group work and communication skills were challenged and required some self-reflection and growth. My partner did not have a good grasp of the assignment and we didn't realize that until it was time to put our work together. In the end, the communication breakdown caused a lot more work for both of us. This issue could have been avoided by a more thorough collaboration process. It also made me consider the importance of clarity when creating instructional materials in my own practice. |
Excerpt From Outline
My Learning Process
When diving into this assignment, I read into it like any other group assignment. It was a 10-12 page paper focusing on program evaluation. We were provided with a fictitious program evaluation that had been completed to assess a distance education program and had to assess and critique the evaluation in terms of how well it was implemented. The assignment was done with a partner who I didn’t know and hadn’t worked with before. This person was a school administrator and seemed well educated and articulate based on our first text based interactions. I immediately assumed they wouldn’t have issues with the assignment and would be a ‘good’ partner whose work I wouldn’t have to overly correct or edit. This assumption was the root of my issue with this assignment and gave me reason to reflect on my own biases and downfalls when working as a team (1.1, 1.2).
My partner and I made our initial contact through Moodle and text, as those were the most accessible for both of us. We decided to do a brainstorm session prior to writing each of our sections of the paper. Our initial text exchanges were clear and concise, and we ended up using a Google Doc to collaborate and write out the main ideas for the paper (3.1, 3.2). We chose Google Docs because we both had experience with it, and we wouldn’t have to spend time learning a new program. Google Docs is also free to use, which makes it appealing for student work (3.5). Once we had the main points down in our outline, we divvied up the sections and went to work. We each chose one of the more basic sections (introduction and conclusion) and then split the more detailed sections including which of Owen’s Forms was used, if the Form was applied well, and other details on the Form application. Once we agreed who would write each portion of the paper, we both individually went to work. I used Word to create my document, because it has more features than the online Google Docs version, but remains highly compatible to transfer between the full program with simple copy and paste. At the end of our deadline, we each shared our writing on a shared Google Doc, for the purpose of making it cohesive and editing each other’s work. Completing my portion of the assignment was straightforward, including navigating Owen’s Forms, researching related literature, and analyzing the case study and relevant program evaluation samples (5.2, 5.7).
When I look back at our outline, I can see where we ran into some trouble, as the outline didn’t really go deep into the details of the analysis. In fact, it was quite basic and general. What I didn’t realize at the time, was that my partner had missed the critique portion of the paper and was focusing on simply describing and explaining what the evaluation had covered.
Luckily, we had given ourselves enough time to edit each other’s work before the final due date. On the date we chose to compile our sections together, I read my partner’s portion and found it was significantly lacking in very important details and critical assessment (1.1, 1.5). While they did a decent job of describing the evaluation case study, they did not evaluate how well the evaluation had been executed (1.2). This was a vital part of the assignment. I panicked! Firstly, I get uncomfortable when critiquing the work of a peer, and this peer definitely needed critical feedback. This stems from some misguided feeling of guilt that pops up when the power dynamic from a teacher-student relationship is absent. As a teacher I am very comfortable with giving students feedback. They submit assignments to me expecting me to grade and critique their work, and in this realm, I know how to be professional and thoughtful. However, when giving feedback to a peer, or even a professor in my own academic journey, the rules seem somewhat fuzzier. It’s more important to avoid hurting feelings or ego when the person you’re giving advice to is your equal or superior (4.4, 4.5). These are relationships that may play into my professional life and academic opportunities in the future, so more is at stake when giving feedback! Secondly, I panicked because wasn’t sure if we would be able to get the assignment rewritten in time. We had spent most of the allotted time preparing the first draft, expecting minimal editing to be necessary. Lastly, I panicked simply because was angry with myself for not realizing there was an issue sooner and because I recognized that this mistake had created a load of extra work for both of us, in the eleventh hour (1.10).
My partner and I had an emergency meeting using Teams, which we chose because of its easy-to-use format and link to our school accounts (1.1, 1.6). I knew that we should talk face-to-face, or at least have a phone conversation to figure out what happened and what to do next. Writing long text messages back and forth would be too time consuming and has a higher risk of misconstruing subtleties in meaning when communicating important details (3.3, 3.6).
In our meeting we both got clarity on a few things. Mainly, I explained that the assignment was meant to be a meta-evaluation, or an evaluation of the evaluation itself. I also apologized because I had discovered my own pitfalls in being a supportive partner. By not clarifying the assignment together before we got started with our writing, we had let each other down. It was poor use of both of our time because they had spent all the time writing something that now had to be redone. We had a thoughtful and kind discussion about the breakdown and made a plan to move forward. I added the necessary details and removed redundancies while my partner got busy with editing (1.12, 1.13). In the end, we were still able to hand in the assignment on time and scored well, but it was a lesson in what cohesive group work really looks like, and what it doesn’t look like (4.9).
One other topic that came up in my conversation with my partner was the importance of clear assignment details and instructions for the sake of the learner. In my own practice it was a reminder to follow good instructional design guidelines, such as relevant academic language for the lessons, examples of good workmanship, and a clear rubric (2.8, 6.2). Good instructional design sets the stage for student success.
Ultimately, I know that I have strong writing skills, and can create purposeful and coherent documents. In this case I co-authored a meta-evaluation of a simulated educational program evaluation (2.3, 4.1). What I missed at the beginning of this assignment was collaborating and supporting my partner’s learning as well. Once I realized my mistake, I was able to connect with them using appropriate technology, and ensure they understood the nature of the assignment and together we were able to create steps to move forward and still create an effective and relevant paper through a process of constructive feedback and supportive collaboration (4.4, 4.5). We had to step back and re-write a large section of our paper to include everything required by the assignment and work more cohesively than we were expecting to, with the aid of communication technology such as video and voice conferencing. In my past group projects for this program, divide and conquer had been the main strategy, but this experience was an excellent lesson in what true collaboration and cooperation should look like. I came to understand the importance of ensuring all team members are on the same page and equipped to handle their duties within a project early on, and how regularly checking in throughout the process is vital to everyone’s success (4.6, 4.7).
Competencies
1. Problem Solving, Analysis, & Reflective Decision Making
1.1, 1.2, 1.5, 1.6, 1.12, 1.13
2. Instructional Design & Development for Equity
2.3, 2.8
3. Communication Technologies and Networking
3.1, 3.2, 3.3, 3.5, 3.6
4. Communication & Interpersonal Skills
4.1, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9
5. Research
5.2, 5.7
6. Management, Organization and Leadership
6.2
Moving Forward
This group assignment was an important learning lesson for me because I always considered myself to have strong teamwork and leadership skills. Getting to the final stages of an assignment and realizing that someone misread the instructions felt like a failure for me. I came to appreciate that being a leader means I can’t assume any part of my team understands simply because they are well educated or articulate. Being a good team member and leader involves regular check ins, asking appropriate questions to ensure understanding, and most definitely requires the ability to effectively provide critical feedback to everyone, regardless of their status. In addition, this experience made me consider the importance of clear assignment guidelines for students, because although this assignment was labeled as a critique, it could have been made clear that this was indeed a meta-evaluation, and that might have lessened the confusion for multiple students, not just my partner. Assignment instructions should be written clearly, concisely, and with the vocabulary that students are learning about in the course. This helps to create connections and encourage contextualization of learning materials. Finally, a detailed rubric indicating all the required components of an assignment gives students a place to check their work and self-assess prior to submitting for summative feedback. This experience grounded my belief and understanding of how instructional materials can be designed intentionally to help students succeed. |
Comments
Meghan Hann
16 October 2023, 3:39 PM
Kim-
This is a great piece and I can hear you telling the story. It is about 650 words in the main body of writing and might need to be expanded upon.
Suggestions:
“the power dynamic from a teacher-student relationship is absent”.- talk more about this!
“Ninth hour” or do you mean 11th?
emergency meeting using Teams,- for you and the partner or with the professor?
“We both apologized and made a plan to move forward. I added details and removed redundancies. He edited and edited again.”- short- staccato sentences. Could be combined for flow or is this with intention?
Explain more on this: “ connect with them directly (although still at a distance)”
In Moving Forward: Repetitive-” a good group”
Great quote- possibly move it higher- it isn’t very visible and it has a core message.
Love the forest background and concise competency lists.
K Hansen
18 October 2023, 10:46 AM
Thanks Meghan... Yes definitely 11th hour, 9th hour is something much different! Ha, thanks for the catch and will work on the rest of your suggestions!
Tamara O'Brien
18 October 2023, 1:48 PM
Kim,
I like the theme that you chose for this narrative in your learning journey. It resonated with me as I read through it. I like that you included the outline as one of your visuals/excerpts. This was informative. Overall the page is aesthetically pleasing. I like the font choice and the visuals!
I can see that you’ve added substantially more than a few days ago, you’re sitting at around 1150 words. That’s a big chunk to add! Three of the main body paragraphs do not have sub-competencies in them. Are there any that you could add here? I think that 4.4-4.9 are applicable throughout this entire section. 4.9 isn’t used yet, and some of your points could relate to what you note for your other #4 competency points.
You don’t use sub-competency 6.1 yet in this however it sounds applicable to the type of assignment you completed. There may be more that could be linked to the assignment, but I know that you have focused on the collaboration process which is both significant and interesting.
Is the “moving forward” paragraph your conclusion? If so, you’ll need to take out the sub-competencies here or move them higher. This is a piece of feedback that Bonnie gave me on my previous artefact as well.
Note on grammar/word choice:
“…and the evaluation was based on John Owen’s evaluation structure, which involves five categories, or Form’s.” – is there supposed to be an apostrophe in the last word?
I enjoyed reading this. You are a good writer and communicate clearly to your audience. I understood your thoughts that you communicated throughout your reflection. Very nice!
Tamara
BonnieM
20 October 2023, 4:20 AM
Hello Kimberly,
Artifact two is coming along nicely, here are some recommendations for improvement.
Considering changing your font size and 18 font size makes your work a bit difficult to read because you have to scroll down the page so much. It also makes any images out of proportion to the rest of the page.
Your introduction is well done.
In paragraph one you will need to add in more context (since your word count is less then 1200 words you have room to add in more details. For example what type of paper were you working on literature review, or research paper? How did you choose your partner?
This person was a school administrator, so I immediately assumed … why did you make this assumption? Did this become a problem? If so why? Once you explain this you can use competencies 1.1 and 1.2 and define your problem.
This would be a whole paragraph in itself.
Then move on to the technology you used. Why did you use google docs? Why did you choose this technology? What were the advantages? Did you consider any sharing technology? What were the advantages or disadvantages of this software? Did you have a conversation about this software? How did it go? What did you learn from collaborating in this way online?
Artifact two is well organized but needs some context examples and details to provide the whole story behind every learning experience. This in turn will provide more sub-competencies. Note the question I provided for paragraph one and apply this process to the rest of the artifact and it will be more detailed and close to completion.
Good job
Bonnie
K Hansen
23 October 2023, 9:21 AM
Thanks everyone for the great feedback, I have added according to your suggestions, I also created a new skin with a similar theme but more regular font sizing. Hopefully this makes it more accessible.
Tamara O'Brien
23 October 2023, 11:11 AM
This is a very attractive page! You did a great job with the theme, color, and font!
I'm looking at the feedback Bonnie gave you and I think that you could even add more context to the last paragraph of your learning process. The sub-competencies that you note look relevant when I checked them on the list but if you could probably add more specific information about how you demonstrated them in this specific experience. That paragraph in particular reads more like a reflection on your learning path, which is great if it's a conclusion but to add some meat to it you could add a few more points...
-"...to create steps to move forward and still create an effective and relevant paper" - what is an effective and relevant paper in this context? What was irrelevant about your writing?
-"Ultimately, I know that I have strong writing skills, and am able to create purposeful and coherent documents" - Seeing your writing, I'm sure this is true! However, because you list two sub-competencies here you will need to say specifically what you did to show this.
-"Ensuring all team members are on the same page, and equipped to handle their duties within a project, and regularly checking in throughout the process is vital to success (4.6, 4.7)." - again, two sub-competencies are listed but the statement is generalized. You do have the option to move the sub-competencies to an earlier place in the artefact reflection where you describe this in more detail.
This page has changed so much and looks even better every time - I love that you're so responsive to feedback.
Tamara
Meghan Hann
25 October 2023, 4:02 PM
Kim-
I like the changes and layout. I can hear the story In your voice and the growing pains are well described. Good work adding some details to your teacher / student dynamic and the feedback hesitation section.
It is coming along very well. :) M
BonnieM
27 October 2023, 4:11 AM
Hello Kimberly,
Artifact 2 is almost complete. After you include Tamara's and Meghan's feedback. Here are a few minor tweaks.
What I didn’t realize at the time was that my partner had missed the critique portion of the paper and was focusing more on just describing and explaining what the evaluation covered. So, how did you manage this situation? did you remind your partner? Was there conflict? What did you learn from the situation?
Paragraph 4 also needs sub-competencies.
Great job
Bonnie
K Hansen
27 October 2023, 1:11 PM
Thank you all for the feedback. I think (hope) we are there!
I added your notes Tamara, including a few more specifics about the general sentences. I added some more competencies to paragraph 4.
The other questions that were asked, I think were implied by my writing. Paragraph 5 explains I managed the situation by meeting with my partner and discussing the issues, then making a plan to complete the assignment. What I learned from the situation is in paragraph 6-8 and the Moving Forward section.
Thanks again!
BonnieM
02 November 2023, 4:55 PM
Hello basically artifact 2 is complete but there is some editing issues for example:
We chose Google Docs because we both had experience and wouldn’t have to spend time learning a new program, and being free to use makes it appealing to student work (3.5).
Please edit the artifact again and then artifact two is complete.
Bonnie