Validating Faculty Advising Through Assessment
Faculty members take on the role of primary advisors on many small campuses. Many report feeling underprepared for the advising role. Assessment of academic advising can raise the awareness and perceived importance of advising and provide helpful feedback for practitioners. We developed a 14-item online advising assessment used to evaluate four important domains of advising: academic advice, advisor availability, advisor as personable and interested, and advising about vocation. We used this assessment to evaluate advisors who participated in an advising workshop designed to enhance their relational and conceptual advising skills. Student evaluations of advisors before and after the workshop showed significant positive differences. We recommend this assessment for advising improvement and as a means of evaluating workshop efficacy.
As the end of the present decade draws near, higher education faces several critical issues. Costs of providing higher education are rising; tuitions are rising; enrollments of four-year institutions are declining (Spence, 1977); many private colleges have closed; the public, legislators, and employers are pressing schools to be accountable for the product they produce; and students want quality education and guidance in reaching their goals in the most efficient manner. Higher education is feeling pressures from many different directions and must try to respond—in some cases the traditional answers will work no longer. As higher education ranks lower on state priority funding lists, academic administrators must find ways to maximize the effects of programs and personnel. (Raskin, 1979, p. 99)
Sound familiar? The opening quotation could very easily refer to the state of higher education today (“2018 Higher Education Industry Outlook,” n.d.) because colleges and universities are experiencing the same challenges they faced almost 40 years ago. In the seminal article “Critical Issue: Faculty Advising,” Raskin (1979) featured the paragraph we used to open this 2019 article. Raskin argued that faculty advisors and the assessment of faculty advising outcomes can maximize the effects of programs and personnel in higher education. Although not a new function, faculty advising remains a difficult area to assess for efficacy and to evaluate for advisor professional development in terms of measurable improvements to the advising process and student success.
Therefore, we address the need for an online advising evaluation instrument useful for advisor professional development and for determining the extent to which training has been integrated into practice. Specifically, we show the influence of a faculty-training workshop on advisor practice as measured by student satisfaction evaluations completed before and after faculty advisor participation in an advising workshop. Our study illuminates the importance of advisor development via workshops and presents an evaluation instrument that can be adapted to institutions of all sizes and types.
Faculty Advising Benefits and Challenges
At many small- and medium-sized colleges, faculty members take on the role of academic advisor. The authors of 2011 National Survey of Academic Advising reported that 83.5% of small colleges (student enrollment of fewer than 5,999 students) enlist faculty advisors (Carlstrom, 2014b), utilize faculty members only (28.2%), or leverage a combination of faculty advisors and professional advisors (56.2%) rather than rely solely on professional advisors (15.5%) (Carlstrom, 2014d). A recent advising survey of various-sized institutions, Driving Toward a Degree (Tyton Partners & Babson Survey Research Group, 2016), indicated that the faculty provided advising at 89% of 4-year public and 93% of 4-year private institutions. Faculty members are assumed to be the campus experts in their particular disciplines, understanding both the courses required to obtain appropriate disciplinary knowledge and the preparation required for employment or graduate work in the field (Hénard & Roseveare, 2012).
Furthermore, most stakeholders of higher education agree that advising should guide students to an applicable breadth of experiences as necessary for their academic and future career accomplishments (Cook, 2013; Hemwall, 2008). For example, Wallace (2014) summarized faculty advising by stating that most respondents to the 2011 NACADA National Survey (faculty, administrators, and professional advisors) viewed faculty advising as an important component of the institution's mission. Therefore, success of students' progression through efficient and effective coursework and the acquisition of proficiencies for their desired career paths have been measured by the advising relationship (Allen & Smith, 2008; National Survey of Student Engagement [NSSE], 2019). For example, Gallup has researched and developed characteristics of effective advisors, which include a mission to help students reach their full potential, rapport with and enjoyment of working with students, empathy for students' emotional needs, perception of the individual student's strengths and weaknesses, and advocacy for student needs at the college level (as cited in Noel-Levitz, 2011). In addition, surveyed faculty members rated 12 functions of advising as important (grouped as subfunctions of integration, referral, information, individuation, and shared responsibility), but the findings revealed that respondents felt responsibility for 3 functions: integration of overall content and major content and suggestion of referrals related to academics (Allen & Smith, 2008). Other research had indicated that faculty members help student learning both within the classroom and within the advising relationship by assisting students in making important connections across the curriculum to the world outside higher education (Hemwall, 2008) and in preparation for a thriving life after college (Gallup, 2015).
In addition to being correlated with the important and challenging functions associated with academics, faculty advising has been correlated with increased retention, student satisfaction, and student success (Brown, 2008; Hemwall, 2008; Tuttle, 2000). Although not addressing faculty advising specifically, Tinto (2010) went as far as saying that advising can make the difference between staying in school and leaving it, and Light (2001) suggested that advising can be the single most important factor in college success. Despite the positive impact of advising, in general, faculty advisors face unique challenges. For example, Wallace (2014) pointed to numerous problems that can affect the quality of faculty advising, namely, lack of understanding, recognition, professional development on advising, support, and time. Moreover, some faculty members perceive themselves as unqualified for the task of advising, considering it more of a counseling role than an opportunity for teaching and learning (Hemwall, 2008), with most receiving no formal professional development in advising during their graduate school experiences.
Determining Advising Effectiveness
Institutional administrators learn about the advising experience by student evaluations of it (Teasley & Buchanan, 2013). Research has shown that the mere assessment of academic advising highlights the significance of advising on campus and reveals the connection of advising to advisor professional development (Cuseo, 2003). Furthermore, leadership at any institution wishing to improve advising resources for students and to create a culture that values academic advising undertakes advising assessment (Cuseo, 2008; Teasley & Buchanan, 2013).
To determine the outcomes of any practice and advance discussion of advising, everyone involved must understand the terminology used to explain the process and the outcomes. For this study, we adhered to the definition stated by Robbins and Zarges (2011): Evaluation addresses information related to the individual advisor, and assessment is focused on the advising program, such as the effectiveness of advisor professional development. We also referred to recent efforts to ascertain advising outcomes. For example, NSSE included topical elements in a national survey. In 2014, of all the survey modules available through NSSE, that of academic advising was administered the most, with 30% of the institutions using it to assess advising (NSSE, 2014). In addition, academic advising evaluations, in the form of student satisfaction surveys, were reportedly used at 35.8% of small colleges (Carlstrom, 2014a), and more institutions with institutional mission statements used assessment for academic advising, with the mission statements serving as proxies for student learning outcomes. The percentage of small colleges that use student satisfaction surveys of advising is small (Carlstrom, 2014a). Because faculty members at small institutions are expected to advise students (Carlstrom, 2014b), we can conclude that few faculty members at small colleges are evaluated as advisors through student feedback.
Of the few institutions that use student evaluations to assess advising, even fewer use them to determine advisor effectiveness. Many advisors from the small institutions surveyed reported advising as a minor consideration in tenure and promotion (22.8%), and when asked to choose from a list of advising rewards, the most endorsed response from advisors at small colleges was “not rewarded” (48.7%) (Carlstrom, 2014c). At the institution where we conducted the study presented herein, a disconnect characterized the report of advising being a minor consideration for rewarded practice and the provost's call to “elevate [academic advising] to a position of fundamental importance in assessing one's work as a professor” (Carpenter, 2003). This misalignment of advising assessment at the small university studied was to provide information for consideration during the reappointment and tenure process because academic advising was featured as one of the four major responsibilities of faculty members: teaching, scholarship, advising, and service.
At the study institution, faculty members are evaluated by their departments, the Professional Status Committee, the president, and the board of trustees three times during the 7-year tenure process (two reappointment decisions and one tenure decision). With each evaluation, the faculty member submits a dossier that includes evidence of successful work in teaching (e.g., course evaluations and class PowerPoint presentations), scholarship (e.g., publications and grants), advising (e.g., advising evaluations), and service (e.g., list of college, disciplinary, and community service activities). He or she also provides a self-evaluation and a teaching philosophy statement. Advising evaluations are administered only for faculty members being reviewed for reappointment and tenure. With this study, we sought to create a more succinct and robust advisor evaluation that could be administered such that all faculty each year could receive feedback designed to improve advising and the student experience at the institution.
Professional Development Workshops for Faculty Advisors
Most small colleges featuring faculty advising (81.7%) offer workshops for faculty advisor development (Carlstrom, 2014d). However, Tuttle (2000) reported that faculty workshops do not address comprehensive advising issues and instead center on factual information. Furthermore, in “Critical Concepts in Advisor Training and Development,” Brown (2008) suggested that advisor development workshops include conceptual components that the advisor needs for understanding the advisees and informational factors regarding the goals of the institution. In addition, workshops should include relational elements that teach advisors to help students with academic and career goal setting. Effective workshops encourage advisors to engage in the advising relationship as a form of teaching to help advisees understand the steps necessary for success in college and identify the skills and interests related to career pursuits (Brown, 2008; Wallace & Wallace, 2015; Wiseman & Messitt, 2010). Despite the importance of faculty advisor communication with students, the impact of advisor professional development, as determined by outcomes of student evaluations of advising, remains unexamined; instead, outcomes are determined on the basis of an advisor's evaluation of the workshop as a participant (Wiseman & Messitt, 2010).
We sought to measure the positive influence of a faculty advisor with the intent of adding to the scant research literature on faculty advisor training and workshop efficacy. Furthermore, we created and administered a tested student satisfaction survey to garner the information necessary for the assessment.
Method
Scale Development for Student Survey of Faculty Advising
The Academic Advising Committee, of which we were members, at the study institution, a small (less than 5,000 enrollments) regional college with 230 faculty members, met to determine the most important domains of advising associated with student success and retention. The committee members read through Cuseo's (2008) chapter “Assessing Advisor Effectiveness” and designed survey items that aligned with the goals of the college. They used the items to measure multiple outcomes for students: affective (perceptions), behavioral (use of campus services), and cognitive (knowledge of academic courses and career objectives), thereby demonstrating construct validity.
We used the four domains determined by the committee to design a student survey of advising. First, we looked at the faculty advisor role in selecting appropriate courses for the student's major and graduation requirements. Second, at the study institution, the concept of vocation—“The lifelong process of tuning into God's call, understanding who we are in Christ, and living our lives accordingly”—undergirds the educational framework; therefore, we examined the extent to which a student felt that this vocation and related career plans were addressed by the advisor. Third, we desired to evaluate the availability of the faculty advisors to students. Finally, the student's perception of the advisor's approachability and interest in students, as linked to higher advising satisfaction and college persistence (Brown, 2008; Cuseo, 2008), was investigated. According to this research outline, we labeled the important domains academic advice, advisor availability, advisor as personable and interested, and advising on vocation. Although the chapter “Assessing Advisor Effectiveness” (Cuseo, 2008) provided the main impetus for determining the advising assessment domains, we now realize that they also fit into the Gallup description of effective advisors that appeared in Noel-Levitz (2011). On the basis of the Gallup descriptions, we connected academic advice with advocacy for student needs at the college level; we aligned advisors as personable and interested in rapport with and enjoyment of working with students, together with empathy for the students' emotional needs; and we looked to assess advising on vocation with the mission to help students reach their full potential according to each individual student's perceptions of strengths and weaknesses.
We also examined an academic advisor evaluation that had been used at the institution primarily for the purposes of reappointment and tenure. We wanted to discern whether those past evaluation items aligned with the domains identified by the Academic Advising Committee. We found that many of the previous evaluation items easily could be added into one of the four domains. Therefore, using committee consensus, we generated 20 items to measure the four advising domains.
Using the survey we thus developed, students indicated the degree to which they agreed or disagreed with each statement about their advising experiences according to a 6-point Likert-type scale. In late April 2012, after Academic Advising Days (two days devoted to advising during which no classes are scheduled), the entire student body was invited to participate in the pilot study by completing the Academic Advising Survey via an e-mail link to a confidential Qualtrics survey. Students provided informed, voluntary consent before participation. A total of 2,036 students completed the pilot survey, representing 57.1% of the student body.
Of the pilot survey respondents, 59.5% identified as female; 16.2% claimed status as African American, Hispanic, Native American, or Asian; and 7.7% reported as international students. We entered the survey data into SPSS and conducted a principal component analysis (PCA) for data reduction of scale items and subscale identification. Based on the items we selected for this pilot survey, we anticipated that the responses might load on the four domains identified. Table 1 shows the component loadings of the PCA. The items that cluster on the same component suggest that Component 1 represents advising on vocation, Component 2 represents advisor as personable and interested, Component 3 aligns with academic advice, and Component 4 was connected to advisor availability. The scale reliabilities per Cronbach's alpha for the domains were strong (as per Lance, Butts, & Michels, 2006): academic advice, α = .91; personable and interested, α = .90; advising for vocation, α = .93; and advisor availability (1 item).
In the Academic Advising Committee report to the administration of the study institution, we recommended that the Academic Advising Assessment, as refined after the pilot data were analyzed, be administered each Fall to all students for all advisors. In our report, we suggested that the feedback from this evaluation would provide individual advisors information about specific student reactions to their advising such that advisors could discern areas for improvement in their practice. We also pointed out that the feedback gives the study institution valuable information about the one-to-one connections students make with their faculty advisors, which may prove helpful in the promotion, tenure, and posttenure process of faculty members.
Final Instrument: Academic Advising Assessment
The academic deans of the study institution requested that the final survey to students be shortened to ensure a high student response rate and that the report be limited to one page for ease of faculty and dean viewing, who must read all such reports in a timely fashion. Therefore, we reworked the items for a better, more succinct fit into the model, resulting in 14 items with strong reliability in each of the four domains, a criterion especially important for short surveys. In this new, shorter assessment tool, we included an item requested by the administration: “My advisor gives me as much time as I need when we meet.” This item was positioned under the advisor as available domain despite it being a better proxy for responsiveness than for availability (the inclusion of it resulted in a lower reliability rating [Cronbach's α = .65] than might have been obtained without it).
To discern test-retest analysis, we administered the Academic Advising Survey consistently for three years, starting in 2014, with an average response rate of 56.9%. The combined data revealed that the advising domains were rated in the following order, from highest to lowest importance: advisor availability, academic advice, advisor as personable and interested, and advising for vocation. The finalized 14 items used in the Academic Advising Assessment, based on the initial PCA and recommendations by the administration, can be found, together with the reliability coefficients, in Table 2.
A student's evaluation report is shown in the appendix. It shows the average ratings of all advisees who rated a particular advisor, and the advisor's academic department and college are included for comparison. The institutional review board (IRB) granted approval for this study.
Academic Advising Workshop
The Lunch & Learn advisor workshop was designed to help the advisors master basics, such as information important to the institution and ways to achieve the educational goals of the student, by providing resources regarding the institution, decision making, and managing student information (Wallace & Wallace, 2015). Therefore, in developing the content, we kept in mind some of the elements discussed by Brown (2008), namely, the importance of conceptual, relational, and institutional information. We included conceptual components that the advisor needs for understanding students and emerging adulthood development. We discussed relational elements to help advisors engage students in academic and career goal setting by including information about decision making and a career self-assessment platform. We specifically addressed the following vocation-related goals for students: understand the holistic concept of vocation; articulate one's own strengths, interests, and values; develop realistic goals; and investigate career options and delineate paths to reach them.
The study institution has recently partnered with a talent and career analytics software company to introduce an online career assessment tool that provides information about an individual's personality, interests, values, workplace preferences, and potential job matches, as identified via O*Net, to guide students into meaningful careers. The talent and career analytics software provides science- and evidence-based tools and processes to match talent and traits with occupations, jobs, and organizational cultures. To that end, career center counselors assist many students, but faculty advisors may be uniquely positioned to help students with career matching because of the relationships they have established with students and their in-depth understanding of their own and related disciplines. Although the information produced by this tool is invaluable, connecting the dots between self-understanding and vocation challenges some students.
Therefore, through the workshop, we sought to deepen faculty advisors' understanding and appreciation of the social and cognitive development of emerging adults, identify best practices associated with vocational advising and coaching with emerging adults, and create their capacity to provide advisees individualized tool assessment interpretation and feedback that promotes more effective vocational discernment. The sessions equipped advisors to deliver a high-quality and consistent experience during student vocational exploration. The workshop consisted of information about vocation definition (as per the study institution), emerging adulthood development (Arnett, 2000), decision-making strategies (Santos, Ferreira, & Gonçalves, 2014), talent and career analytics platforms, appreciative advising tips (Bloom, Hutson, & He, 2008), and information about the Career Center. At the conclusion of the workshop, participants were given an opportunity to ask questions.
A total of 52 individuals (faculty members [n = 27], administrators [n = 3], Student Success Center staff members [n = 15], Career Center staff members [n = 7]) participated in one of three 1.5-hour Lunch & Learn workshops offered during the Fall semester of 2016. Twenty-two faculty participants (10.6% of the study institution's faculty advisors) provided informed voluntary consent allowing us to examine the scores attributed to them, before and after workshop participation, by their advisees who evaluated them using the Academic Advising Evaluation. All participants received lunch during the workshop and four dining hall tickets to use for meeting with some of their advisees over lunch. The institutional review board approved this study. Before the workshop sessions were facilitated, faculty participants completed a paper-and-pencil survey to provide some baseline information about the frequency with which advisors engaged in career and graduate or professional school discussions and broached issues of self-understanding with advisees.
Immediately after the workshop, participants were asked to complete a survey in which they had the opportunity to identify the three most important points of information they learned from the workshop. After the Fall advising session, a second survey was administered to the faculty advisors who had attended the workshops; they completed an online, self-reflective survey in which respondents reported on their personal experiences of advising, specifically on the talent and career analytics software assessments. Participants (100% response rate) rated their level of preparedness to discuss personality, interests, values, workplace preferences, and the O*Net job match before the advising session (1–5 on a Likert-type scale, with 5 indicating the highest level of perceived preparation). They also rated their confidence during the advising sessions when discussing each of the talent and career analytics software assessment results (1–5 on a Likert-type scale, with 5 indicating the highest level of confidence).
Results
In the survey administered before the Lunch & Learn advisor workshops, the 22 faculty advisor participants reported that they discussed careers and graduate school very frequently (25.0%), frequently (46.0%), rarely (25.0%), and never (3.6%) and self-understanding very frequently (14.3%), frequently (39.7%), occasionally (35.7%), and rarely (10.7%). Immediately after the workshop, they named the most important information learned as follows: emerging adulthood brain and cognitive development (22.9%), talent and career analytics software background (20%), decision and indecision characteristics (17.1%), talent and career analytics software tools and resources (11.4%), vocation definition (11.4%), Career Center resources (8.6%), talent and career analytics software usage tips (5.7%), and appreciative advising tips (2.9%).
The Academic Advising Assessment scores given by students in 2015 (N = 236) and 2016 (N = 246), before and after the faculty advisors participated in the workshop, were analyzed. Specifically, we conducted a repeated measures multiple analysis of variance (MANOVA), and we found that students attributed significantly higher Assessment scores to faculty advisors after the workshop: F(5, 17) = 2.76, p < .05; Wilks' Λ = 0.55; partial η2 = .45. This very large effect size (as per Lakens, 2013) indicates a strong effect of the workshop on the faculty advisor evaluations. Planned post hoc Bonferroni analyses indicated that the significantly improved measure in 2016 was “personable and interested” (see Figure 1).
Citation: NACADA Journal 39, 1; 10.12930/NACADA-17-034
We compared the Academic Advising Assessment scores of all the faculty advisors from 2015 to 2016 (n = 208) without the data from the 22 faculty advisors. In the repeated measures MANOVA, we found significantly higher Assessment scores for all college faculty advisors in 2016 than in 2015, F(5, 203) = 8.30, p < .001; Wilke' Λ = 0.83; partial η2 = .17, a moderately large effect (as per Lakens, 2013). Planned post hoc Bonferroni analyses showed that the significantly improved measure in 2016 was “personable and interested” (p = .02) (see Figure 2).
Citation: NACADA Journal 39, 1; 10.12930/NACADA-17-034
The workshop participant mean total score (n = 22) for the Academic Advising Assessment administered in 2015 was 5.36, and for 2016, it was 5.45. For faculty advisors who did not participate in the workshop (n = 208), the 2015 Academic Advising Assessment mean score was 5.32, and for 2016, it was 5.33. Although the overall mean ratings for faculty workshop participants were higher in 2016 than they were for all faculty advisors at the college for the same year, these differences did not reach statistical significance, nor was there an interaction effect in the repeated measures MANOVA.
According to the end-of-semester online postworkshop survey, the faculty participants (N = 22) responded about whether they discussed the talent and career analytics software with students. Of the faculty advisors, 68% reported that they discussed feeling confident in their preparation to discuss personality measures and expressed confidence while discussing these areas with students. Of the faculty advisors, 32% reported discussions about interests and workplace preferences, and 27% reported talking about values and the O*Net job matching with students (see Figure 3).
Citation: NACADA Journal 39, 1; 10.12930/NACADA-17-034
Discussion
Through this research, we provide results on a topic that is rarely studied: the impact of a faculty advisor workshop on student satisfaction evaluations of advising (Allen & Smith, 2008). Using the 14-item Academic Advising Assessment instrument designed specifically for this study, we evaluated whether advisor participation in a workshop that addressed areas important to the institutional mission–emerging adulthood development, decision strategies, talent and career analytics software, Career Center resources, the study institution's vocation definition, and appreciative advising tips—was associated with higher student assessment scores regarding advising. The students' scores for advising for faculty members who attended the workshop were significantly higher than they had been for the same advisors the previous year overall and especially within the domain of advisor as personable and interested. Therefore, our results indicate that the brief evaluation of student perceptions of advising can be used to assess the efficacy of an advisor workshop.
Workshop Purpose and Focus
Our workshop provided information about professional development in congruence with the discussion of Brown (2008): conceptual, informational, and relational advising. First, the workshop included guidance on conceptual elements of emerging adulthood development (Arnett, 2000) and career decision making, through models of indecisiveness (Santos et al., 2014), for advisors to better understand students and the conceptual elements important to the study institution, such as vocation. The emphasis placed on understanding the student aligned with recommendations from Hemwall and Trachte (2005), who discussed the way learning takes place when advisors understand the social context and needs of the individual student. Second, the workshop participants learned about delivery of informational elements through approaches such as appreciative advising (Bloom et al., 2008). Third, relational elements were presented within the context of the career assessment platform; specifically, the approach for addressing the talent and career analytics software with students was explained.
In a follow-up survey, the participants rated the top three areas of learning during the workshop as emerging adulthood development, talent and career analytics software, and career decision making. These areas were defined as conceptual and relational, as several authors featured in Academic Advising: A Comprehensive Handbook (Gordon, Habley, & Grites, 2008) had suggested were the most important elements in advising; these areas of practice distinguish advising from the informational, clerical activities that do not embody advising practice. Despite knowing the reported top workshop learning areas, we cannot determine whether participants felt satisfied with the level of knowledge gained or whether they believe these areas reflect those most important for advising.
Student Assessment of Faculty Advising
The aggregated group of faculty advisors who participated in the workshop received higher overall advising Assessment scores than they had received the previous year. This remarkable, measurable improvement in scores highlights the importance of training for faculty advisors and the evaluation of those efforts as cited in the advising literature (Aiken-Wisniewski, Smith, & Troxel, 2010; Cuseo, 2008; Wiseman & Messitt, 2010). We recommend, especially for institutions at which advisors already undergo evaluations, an examination of the advising assessments both before and after professional development workshop participation (per Cuseo, 2008) to assess the efficacy of the workshop and implementation of the principles learned. Furthermore, the numerical results of improvement can be used to promote the importance of training and learning about advising to the faculty and administration.
Student evaluation scores on the domain of advisor as personable and interested increased the most for the faculty advisors who participated in the workshop and for those at the study institution for whom data were available. Although not the highest rated domain by students on the Academic Advising Assessment, the advisor as personable and interested domain scores consistently increased for advisors at the study institution from 2014 to 2016. This notable improvement indicates that students perceive that advisors care about them, and hence, students may receive advisor recommendations favorably (Brown, 2008; Raskin, 1979; Wiseman & Messitt, 2010). Likewise, the characteristics of effective advisors, as developed by Gallup, could be categorized in the advisor as personable and interested domain: embracing the mission to help students reach their full potential, exhibiting rapport and enjoyment when working with students, showing empathy for students' emotional needs, perceiving the individual student's strengths and weaknesses, and advocating for student needs at the college level (Noel-Levitz, 2011).
The differences in Academic Advising Assessment scores for those who participated in the workshop and those who did not participate failed to reach statistical significance, a finding that might be attributed the unequal size of the two groups: 22 faculty workshop participants who gave permission to evaluate before-and-after scores versus 208 faculty advisors who declined to participate in this workshop but whose student evaluation data were available for analysis. Furthermore, the increase in the advisor as personable and interested domain for both workshop participants and nonparticipants could be the consequence of an overall improvement in the institutional culture concerning advising. Data showed that the advisor as personable and interested scores had increased to 5.37 in 2016 from the baseline of 5.19 in 2014, the first year the shortened evaluation, Academic Advising Assessment, was administered and the institution's leaders increased the emphasis on advising. During this time frame, a number of initiatives at the study institution may have encouraged faculty members to place greater value on the importance of the advising relationship with students. These collegewide initiatives included yearly advisor evaluations, a move of advising from fourth to third place in the faculty responsibilities list (teaching, scholarship, advising, and service), and an annual faculty award for advising and mentoring given at an awards dinner. Perhaps these initiatives raised the level of awareness and importance of advising in the minds and behavior of the faculty advisors, resulting in the outcome of consistently improved scores on advisor as personable and interested and thereby better meeting the needs of students.
Long-Term Benefits of Assessed Advising
Our study raises some important considerations about the importance of the advising process for student success after college. Gallup (2014, 2015) presented investigations into six experiences of college students that lead to life success, and advising may directly affect at least two of these factors: “My professors cared about me as a person,” and “I was encouraged to pursue my goals and dreams by a mentor.” In the advising relationship, the faculty advisor can demonstrate holistic concern for a student in a personable and interested manner. Likewise, as the advisor and student discuss the student's vocational pursuits, the faculty advisor can offer support and reassurance that enhance the student's future life. The advising relationship could indirectly influence some of the other four Gallop-identified factors: “I had at least one professor who made me enthusiastic about learning,” “I worked on a semester-long project,” “I was able to apply my classroom learning via an internship or job,” and “I was really active in extracurricular activities and organizations.” Professors can inspire student enthusiasm in learning and help them find semester-long projects in which to engage. As part of the holistic and career discussion involved in an advising setting, advisors also can encourage relevant internships, jobs, and extracurricular endeavors.
After attending the workshop, faculty advisors reported confidence about discussing some of the career assessments. However, not all workshop participant–advisors discussed the talent and career analytics results with their students. Including discussions of these analytics typically requires advisors to spend additional, substantial time in an advising session, and not all advisors could accommodate the increased appointment time. Furthermore, although they deemed such conversations important for students, some faculty members think that the responsibility for discussions related to in-depth self-understanding and career falls elsewhere (Allen & Smith, 2008).
Measures of Advising Assessment
Overall, our study supports the theory that advising assessment can promote advising improvement at the institutional level (Cuseo, 2003; Teasley & Buchanan, 2013). As summarized by Powers, Carlstrom, and Hughey (2014), research has shown that advising assessment promotes student retention, success, and understanding when advisors receive beneficial feedback from student reports of the advising experience. Furthermore, our study reinforces the importance of reliable and valid advisor evaluation instruments as cited in the literature (Allen & Smith, 2008; McClellan, 2011; Teasley & Buchanan, 2013).
Our assessment was found to be effective in measuring the impact of a workshop training. Although advisor professional development has been found to be minimal at most institutions (Cuseo, 2003), and slightly better at smaller colleges (Carlstrom, 2014a, 2014c, 2014d), our research shows that it may be linked to improvements in advisor Assessment scores, implicating advising relationships as important in connection to student retention (Brown, 2008; Hemwall, 2008; Powers et al., 2014). Despite numerous advisor professional development opportunities in online formats, such as webinars or eTutorials, and summer courses through NACADA's online education (2019), the efficacy of this training on advisor effectiveness is finite.
Limitations
A limitation of our use of the Academic Advising Assessment (primarily student satisfaction measures) to examine the impact of an advisor workshop involves the variety of factors that could influence the student evaluations. For example, advisors who participated in a Lunch & Learn advisor workshop may have been more willing to meet for additional time with their advisees, and students who interact more frequently with their advisors tend to render higher ratings of advising (Hester, 2008).
Although our study provides insight into the reliability and validity of a short online advising assessment and its efficacy in evaluating advisor workshop participation, the Academic Advising Assessment was administered at one regional college of 3,918 students with 230 faculty advisors. Therefore, the results cannot be generalized to other institutions, including other small colleges using primarily faculty advisors. In addition, institutional factors may have influenced the faculty members who chose to attend the professional development workshops; the importance of advising or students' perceptions of faculty members and their advising also might have affected the findings. Faculty members who advise students struggling academically or pursuing a career choice misaligned with their academic ability must engage in difficult conversations. As a result, these faculty advisors may receive lower ratings as a function of students' academic struggles and unrealistic expectations.
Our study was limited in the number of faculty advisors who participated in a Lunch & Learn advisor workshop. We offered three sessions on different days and at different times to accommodate teaching schedules as well as a food incentive; however, faculty participation in the workshops was not as high as we had hoped.
Future Research
Future researchers should examine the relationship between students' grade point averages (GPAs) and the assessments they provide their advisors; for example, do students with high GPAs give their advisors higher ratings than students with low GPAs do? Also, a similar workshop with more faculty participation might lead to important insights about advisor professional development. An interesting future study could examine workshop efficacy between advisors at institutions where advising training is required and those where professional development in advising is treated as optional.
Implications
The findings from our study prompt several recommendations. We suggest that advising workshops focus on the contextual and relational aspects of practice. For example, we recommend targeted learning on emerging adulthood development, career decision-making processes, and student self-understanding as related to evaluations of students' personality, interests, and workplace values. The career assessment platform offers helpful tools for discussing this self-understanding with students, but advisors need to learn the tools and ways to discuss the output with students.
We also recommend assessing the efficacy of these advising workshops by comparing the student evaluations, such as those from the Academic Advising Assessment, for statistically significant differences before and after advisor workshop attendance. Future advancements of this evaluation process should include measures of student learning via the advising context, as recommended by several advising experts (Aiken-Wisniewski et al., 2010; Powers et al., 2014). In this era of focus on higher education assessment, institutional stakeholders must effectively assess their advising program, including an evaluation of advisors, student learning, and the influence of advisor professional development.
The 14-item online tool, Academic Advising Assessment, provides reliable and succinct information for assessment of academic advice, advisor availability, advisor as personable and interested, and advising for vocation. Because the domains we examined have been shown in previous research to be important for feedback to advisors (Cuseo, 2008; Srebnik, 1988), we believe this instrument easily could be adapted by stakeholders at other institutions. Furthermore, the advising assessment measures the relational (personable and interested, availability), informational (academic advice), and conceptual (vocation) elements that were discussed by Brown (2008) in his chapter on advisor training and by Wallace and Wallace (2015) in their chapter on faculty advising. Therefore, we contend that the advising evaluation instrument could be a suitable means for assessing the effectiveness of any workshop on professional development of advisors.
Summary
In conclusion, Hemwall (2008) stated that providing support for faculty advisors in the form of administrative attention to advising, evaluations that offer feedback to both advisors and administrators, and recognition or rewards for faculty advising could help colleges make important steps toward realizing the potential of faculty advising, and this purposeful use of resources proves especially important for new faculty advisors (Wallace & Wallace, 2015). The study institution was continually working on appropriate support for faculty advising through the creation of a faculty governance committee on advising; faculty advising responsibilities outlined in the faculty handbook; the implementation of yearly advisor evaluations read by deans, department chairs, and the advisor; and an advising and mentoring award presented with other faculty awards at the conclusion of the academic year. The result of this attention to faculty advising was reflected in the all-college advisor evaluations, namely, in the relational domain of advisor as personable and interested and the positive impact of advisor professional development on student evaluations of advising. Hemwall and Trachte (2005) asserted the value of the advising context for furthering the institutional mission in conjunction with student learning. Academic advising offers a means of shaping a student's educational experience in preparation for a life of service and a life as an agent of renewal, important outcomes in the mission statement of the study university.
Student responses on Academic Advising Assessment of before (2015) and after (2016) faculty advisor (N = 22) participated in Lunch & Learn advising workshop
Note. Responses on Likert scale (1 = strongly disagree to 6 = strongly agree) of statements on advisors on domain. F (5, 17) = 2.76, p < .05; Wilks' Λ = 0.55; partial η2 = .45; post hoc Bonferroni, *p < .05.
Student responses on Academic Advising Assessment regarding all faculty advisors (N = 208) in 2015 and 2016
Note. Responses on Likert-type scale: 1 = strongly disagree (worst rating) to 6 = strongly agree (best rating) of positive statements about advisors. Repeated measures multiple analysis of variance: F(5, 203) = 8.30, p < .001; Wilks' Λ = 0.83; partial η2 = .17.
*p < .001 (post hoc Bonferroni).
Faculty advisor perceptions of preparation and confidence in advising discussions after attending Lunch & Learn workshop on advising (N = 22)
Note. From postworkshop survey. Scales range from 1 to 5, with 5 indicating the highest levels of preparation and confidence.
Contributor Notes
We express gratitude to the 2017 members of the Academic Advising Committee and Bryan Dik for their helpful comments on an earlier draft of this manuscript. We thank the Center for Social Research, especially Megan S. McNamara and Jordan Daley, for assistance in data gathering. This research was supported by a grant from the Network for Vocation in Undergraduate Education (NetVUE).
Julie E. Yonker received her PhD in cognitive/experimental psychology at Stockholm University, and she is an associate professor of psychology at Calvin College. Her research interests center on positive contributors to college students' growth in academics, self-understanding, and vocational discernment. Her research has been published in journals such as Journal of Adolescence, Psychology of Religion and Spirituality, and Assessment and Evaluation in Higher Education. Yonker served as the chair for the Academic Advising Committee and received the 2017 Calvin College Advising and Mentoring Award. For questions about this article, contact her at julie.yonker@calvin.edu.
Dana Hebreard received her PhD in educational leadership at Western Michigan University. She works as the director of career service at Aquinas College. Hebreard was recently recognized as the 2016 Outstanding Academic Advising Administrator by the Michigan Academic Advising Association, and she received the 2017 Commission & Interest Group Division Service Award from NACADA. She can be reached at dmh005@aquinas.edu.
Brian D. Cawley received his PhD in industrial-organizational psychology from the University of Akron (Ohio), and he is a professor of business at Calvin College. His current research interests include performance feedback, calling/vocation, and cultural agility. Along with serving as an active practitioner, Brian's research has been published in journals such as Journal of Applied Psychology, Organizational Behavior and Human Decision Processes, and Industrial and Organizational Psychology: Perspectives on Science and Practice. He can be reached at bcawley@calvin.edu.
