Part 2: Updated Public Version Of Maker Literacies Student Learning Data

U T A with star in the center, used when staff photo is unavailable

by Martin Wallace
December 21 2018

IMLS logo

[This is the second in a series of blog posts about our data gathering, processing and analysis methodology.]

Happy Holidays!

Because of other priorities, I have not been able to move to the next steps in our data gathering, processing and analysis series. As we drew nearer to the end of the fall 2018 semester, I decided it would be best to wait until all of the fall 2018 data had been collected and aggregated with the original data set before moving forward with processing and analysis.

So now that I’ve collected and aggregated fall 2018 data, here are the updated numbers. Please be sure to read the previous post on this topic, as all disclaimers about the data mentioned there are still applicable, plus you will get some context for the information included below.

Over three semesters (Fall 2017-Fall 2018) 765 students, including students who opted in and who opted out, completed the pre-self-assessment survey. Of these, 717 consented to participate in the study and were issued a corresponding post-self-assessment survey. Students who did not provide consent have been removed from the public data.

Of the 717 students who consented to participate, 346 completed the post-self-assessment survey. Students who did consent to participate and completed the pre-self-assessment survey, but who then did not complete a post-self-assessment survey have also been removed from the data.

The updated data has been posted to Mavs Dataverse but is currently pending review by our Dataverse administrator. I expect it to be approved soon after everyone returns from their holiday breaks the first week of January. It's possible that both the updated and the original data sets will be there, so be sure to use the one labeled Fall 2017-Fall 2018, not the one labeled Fall 2017-Spring 2018 (I might delete to original, if our Dataverse admin recommends doing so).

This data set is now a complete data set. We will no longer be using the surveys from the pilot phase of the Maker Literacies program. Beginning in spring 2019, we’ll begin using a set of fully revised surveys and begin collecting a new data set. As mentioned in the last post on this topic, we are treating this data as our practice data set as we work through the process and improve our methodology for the next phase of the program. Of course, there is potential for mapping some of the data from this set to the next set, and I’ll write more about that when the time comes.

In my next post in this series, I will explain and demonstrate some metrics that we will use to help identify potentially unreliable responses from this data set. The combined use of “intraclass correlation coefficients,” “average absolute deviations,” and survey durations among survey responses will help us to further validate the reliability of results. Once the potentially unreliable results are removed from the data, we can finally perform some analysis of the results.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <button> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.