Evidence

Our evidence consists of many varied sources, including surveys, interviews, focus groups, review of student work, and research studies.

Student Perceptions of ePortfolio

Dr. Linda Anstendig and Political Science/Communications student, Jemma York, completed an undergraduate research study to capture the student perception of ePortfolios, a topic that has not yet been thoroughly examined by the international ePortfolio community. With the help of two student eTerns, they were able to interview a number of students on their ePortfolio experience and write a report on their findings. Jemma York presented the study at the Pleasantville Undergraduate Research Showcase in May. The full study can be viewed on its ePortfolio page: https://eportfolio.pace.edu/view/view.php?t=phFKwQLR9Mylm6sTfctG

Teaching Circle Surveys

Since 2010 we have surveyed the Teaching Circle participants at the end of their TC semester.  This survey attempted to gauge their comfort level with ePortfolio, and how they would use it in class. Trends we have seen since 2010 include the need for more technical support, the desire for more hands-on / computer lab training, and the need expressed by the faculty members themselves that they need to devote more time to practicing and getting comfortable using ePortfolios.

In spring 2013 we distributed a survey to all faculty who completed the ePortfolio teaching circle seminar since fall 2010. Although we only received 21 responses, the results gave us needed insight into the successes and challenges our faculty are experiencing with ePortfolio. Out of the 21 responses, 19 continue to use ePortfolio in their courses. Our survey respondents reported that faculty ask students to do a variety of different things, including:

  • posting a final paper/project
  • creating a journal
  • creating a career-focused ePortfolio
  • posting reflections and posting items that relate to their coursework

18/21 faculty included ePortfolio in 20-50% of the student’s final grade. We found that ePortfolios were being implemented across a variety of disciples, including accounting, English, communications and math.

One faculty member noted that she found ePortfolio builds confidence, self-awareness and levels the playing field since ePortfolio is a new tool for everyone. She also said she felt that the public nature of ePortfolio gives validity and a sense of importance to assignments. We are thrilled to see our faculty making these connections.

C2L Core Survey

Fall 2011 C2L Core Survey Results: We were disappointed by our small sample.  We had only 22 students participate.  We asked for faculty support in distributing the survey, but learned we need to be more direct in how we approach classes asking for their participation in the survey.  We plan on having our eTerns enter selected classes and administer the survey, which will hopefully increase results. One challenge was the timing of these surveys; the end of the semester is a hectic time for faculty and staff alike, therefore, it was challenging to get courses to participate. Given our very small sample, it seems hard to generalize about our entire ePortfolio project . We have approximately 2,000 ePortfolios users.  But these results highlight a few key points:

  • 19 out of the 22 students felt that work in the course emphasized, synthesizing and organizing ideas and applying theories or concepts to practical problem or in new situation, but only 14 of these students agreed that ePortfolios helped them to make connections between ideas.
  • 14 students responded that ePortfolios helped them to make connections between ideas.
  • Only 6 students stated that peers/classmates provided useful feedback.
  • 17 students stated that the instructor provided useful feedback.
  • 17 students enjoyed building their ePortfolio
  • Only 10 students said that ePortfolios were an important part of the course.
  • Only 10 students included information about school activities outside of the classroom.

These results highlight some of our strengths and areas for development.  These results help us to see that we need to encourage more student interactions with ePortfolios to promote feedback.  We hope to encourage more use of ePortfolios for the “informal curriculum” and making connections between all parts of student learning.  And need to continue to work with faculty to make sure that ePortfolios are fully integrated into course work – not just added on.

Fall 2012/Spring 2013 Core Survey Results: Our most recent core survey results show that we are making progress, but we still have more work to do.  They confirm our feelings that we are increasing students understanding of some of the goals of ePortfolio, but these results show more of an understanding of ePortfolios as a synthesizing tool rather than a personal development tool.  Students seem to understand the career potential here rather than the potential to share with family and friends.

These results still show some frustration with the user friendliness of our platform. At the same time we are encouraged by how new students in an online program in Summer 2013 (Pace Prep) were able to use ePortfolios with minimal support.

We are discouraged by the results for question #18 showing that even if students value ePortfolios, they aren’t enjoying it.  This response – as well as the results from #24 and #25 – indicate that our students may not be seeing the “big picture” of ePortfolio and may see it only as course tool.  This combined with the varied responses in the open ended questions shows us that ePortfolios are being incorporated in a variety of ways and with a range of effectiveness.

These results raise many questions for us including:

  • How can we do next generation faculty development to increase faculty understanding of ePortfolios?
  • How can we do more students awareness work to help students understand the benefits of ePortfolios beyond a single course?
  • How can we address some of the perceived user interface issues?  How can we utilize our eTerns to support this effort?

Tenure and Promotion ePortfolio Survey


Beginning in Fall 2011, faculty applying for Tenure and Promotion were required to use ePortfolios to display their eDossiers. Our surveys from both candidates and reviewers have yielded largely positive results about the process. There were 18 faculty going for Tenure and Promotion (TAP) and 72 Reviewers. Fourteen of the faculty applying for TAP and 35 of the Reviewers returned the survey.

Seventy-nine percent (79%) of the Reviewers who responded did not attend any training sessions. Of the Reviewers who took the training session and responded, 61% were either satisfied or very satisfied with the training. The responding Reviewers also found it easy or very easy for navigating the platform. Eighty-seven percent (87%) of the responding Reviewers felt the e-portfolio platform showcased the dossiers well; 77% found the platform easy to navigate to some degree. Ninety-three percent (93%) stated that the University should use the same platform for next year’s tenure and promotion.

Fifty percent (50%) of the faculty applying for TAP who responded took the training session with the majority, 76%, finding the sessions satisfying. Over 80% of the respondents were satisfied working with the e-portfolio platform; over 70% found the uploading of documents easy. Eighty-five percent of the faculty applying for TAP felt that the e-portfolio platform showcased their dossier well; close to 90% stated the e-portfolio platform should be used next year. Written suggestions included:

1. Make a training video
2. More training sessions are needed in September and less in late October and November.
3. Forced organization of the dossier saving time and paper copies
4. Some concern that local TAP committee members were not familiar with the new process
5. Took “way too long” to separately review each attachment

The School of Education and College of Health Professions have also chosen ePortfolios to performance review evaluations as a means to assess faculty work.

ePortfolio for Assessing Core Curriculum Learning Outcomes

Since spring 2011 we have conducted three multidisciplinary assessment pilots using ePortfolios to evaluate evidence of three Core Curriculum learning outcomes:  Written Communication, Analysis, and Information Literacy and Research Skills.

When reviewers’ grades and comments are compiled and compared over the 2011 to 2012 pilots, student competency in the Analysis outcome is fair, the Written Communication outcome is fair-to-good, and competency in Information Literacy was fair-to-good in 2011 but is only fair in 2012. NOTE: The 2011 assessment is based on the work of only 72 out of 176 students, while the 2012 assessment is of 68 students out of 162 on the rosters. In both administrations, valuable information about Core outcomes was offset by a lack of accessible student work. Less than half of the students had work available to review because they had failed to set their permissions to allow viewing by Pace users other than their friends and teachers. Furthermore, many of the 72 ePs that were viewable in Spring 2011 did not contain work of sufficient academic quality for evaluation by the rubrics.

In Spring 2013, When reviewers’ grades and comments were compiled, student competency in the Written Communication outcome was good, and competency in Analysis and Information Literacy was fair-to-good.

2013 ePortfolio grades by rubric (four-point scale):

Analysis

Information Literacy

Written Communication

count

36

count

32

count

42
Identification 3.08 Determine extent of info needed 2.68 Content and development 3.19
Inference 2.72 Access and retrieve info 2.71 Organization 3.30
Analysis 3.11 Evaluate info, sources critically 2.55 Style 3.09
Conclusion 2.61 Access and use ethically 2.39 Mechanics 3.09
Overall 2.88 Overall 2.58 Overall 3.17

This assessment is based on the work of 110 students out of 165 on the rosters. Compared to the Spring 2011 and 2012 review sessions, student work was more accessible to reviewers—approximately two-thirds were available versus less than half in previous years.

Over the course of the three pilots, reviewers had the opportunity to reflect upon their experiences using the rubrics to grade student work. In response to the prompt “What did/didn’t work overall?” responses included these comments:

–          Reviewers voiced their frustration with student permissions not being set correctly
–          Some noted that all assignment types were not adequately addressed by the rubrics. For example, all criteria of the Written Communication rubric do not apply to blog entries, and the Information Literacy rubric does not address academic dis/honesty or field research.
–          In 2013, reviewers still frequently noted their lack of access to student work, but also noted their comfort with the rubrics.
–          Reviewers reported general satisfaction with students’ reflective statements, but also noted that many reflections were general; two reviewers suggested the need for separate rubrics for reflections.
–          Some reviewers noted that student improvement was evident in ePs containing multiple semesters of work.
–          Written Communication reviewers reported satisfaction with the rubric, while Information Literacy reviewers noted that student work did not satisfactorily meet many of the standards of the rubric. Analysis reviewers also noted that student work did not evenly meet all dimensions of the rubric, and one reviewer suggested that providing more guidance to students in the analysis outcome would be beneficial to their writing.
–          In 2013, the eight who completed the reflection tab reported spending about 20 minutes reviewing each ePortfolio, with Written Communication reviewers needing the least amount of time, 10 to 15 minutes, while Analysis reviewers needed the most time, about 25 minutes per ePortfolio.
–          Many reviewers noted that the variety of student work was interesting but not always easy to evaluate with their assigned rubrics.
–          Some reviewers offered suggestions for future ePortfolio assessment projects—one reviewer suggested including a course name and description for each student portfolio, and another reviewer recommended further instruction for faculty on permission setting, student reflections, and the importance of uploading analytical papers.

In fall 2011 we also began an assessment pilot with the Pleasantville campus English Department writing faculty in partnership with instructional librarians to assess students’ Written Communication and Information Literacy competencies via ePortfolios, using similar rubrics.

Writing Faculty and librarian reviewers also had an opportunity to reflect on the pilot and contribute to our understanding of how we can improve this process. In response to the prompt “What did/didn’t work overall?” in 2012, reviewers cited the ease and convenience of viewing student work electronically, difficulty entering comments into the Excel rubric grading sheet, and dissatisfaction with the whole-integer (per dimension) grading options. The majority of the reviewers reported that students’ reflective statements were uneven in quality: some were thoughtful and some were superficial. Reviewers’ general comments about their reviewing experiences included satisfaction viewing the variety of student work available in the electronic portfolio, comments about the availability of student work, and suggestions for improving the setting of permissions.

“Evaluation of the usage of E-Portfolio by Pace University students and their views of its utility in their academic and professional development: A Graduate ePortfolio Study”
In the spring 2012 semester, Pace graduate student Ikechi C. Okoronkwo chose to conduct a comprehensive study on ePortfolio usage at Pace for his MBA research methodology course. After administering a survey to all ePortfolio student users, his sample consisted of 185 responses. He reached the following conclusions:

Freshman and graduate students use ePortfolio most frequently, leaving sophomores and juniors as infrequent users.  Graduate students use ePortfolio for professional reasons, which makes sense because they would be searching for careers.

Ikechi states that “of the students that have actually accessed their E-Portfolio account, 81% were compelled to do so because it was mandatory for a class.”  This means that the way to influence students is to make it mandatory.

It seems as if students are not taking advantage of the tool on their own.  They complain that Pace is not preparing them for life after graduation, but have not yet taken advantage of all that Pace has to offer.  Students need to be motivated to use the site and they need to understand the professional use of ePortfolio.  Ikechi says that “it is not perceived as highly beneficial to their professional development,” which is something that students need to realize in order to reap the full benefit.

Major Findings:

  • Graduate students account for the highest percentage of Pace University students who actually use their E-Portfolio accounts.
  • Graduate students and freshmen account for the highest percentage of Pace University students who state that they have a passion or talent outside of their field of study that they could use to pursue a career.
  • No freshmen have shown their E-Portfolio in a professional capacity and of the other student statuses who use their E-Portfolio account and have shown their E-Portfolio to someone else in a professional capacity the breakdown is 18% of sophomores, 13% of juniors, 17% of seniors and 30% of graduate students.
  • 70% of all respondents state that they have a passion or talent outside their field of study however 41% will only apply for jobs within their field of study after graduation.
  • 26% of the respondents do not feel that Pace University has prepared them for life after graduation however only 46% of this sample are using the E-Portfolio access that is provided to them by Pace University.
  • 81% of all E-Portfolio users were compelled to do so because it was mandatory for a class but only 43% feel that their use has improved their academic performance.
  • With 76% of all respondents stating that they view E-Portfolio as an effective management tool for a person’s online personal brand, only 54% actually use the site and only 19% of those users took a personal initiative to use the E-Portfolio site without a faculty requirement.
  • In general, students are comfortable using the site and find it easy to use. However, it is not perceived as highly beneficial to their professional development.”

ePortfolios and Retention

In fall 2013 we began preliminary work on establishing a connection between ePortfolio usage and retention. With the help of Pace’s Office for Planning, Assessment and Institutional Research, we were able to compare NSSE data with our ePortfolio data. As a result, we found a modest positive correlation between the group of freshmen who used the ePortfolio within their first academic year and their retention. The document below shows these findings. We plan to continue this research as our ePortfolio reports become more robust.

Fall 2013 Retention Data: retention data fall 2013-edited

Correlations between ePortfolio use and NSSE results 2011 through 2013: NSSE summary jan 2014(KDrev)