Assessment & Analytics

Christina Hennessey Kevin Phillips (Unlicensed)

ULMS Analytics & Assessment Functional Committee met Wednesday, June 8, 2022 11am-noon

Attendees

Christina Hennessey (Chair/CSUN), Jung Ah Lee (SJSU), Shonn Haren (CPP), Kevin Phillips (Vice-Chair/CSUF), 

Absent: Lisa Roberts (CSUS), Brandon Dudley (CO), Patrick Flanigan (SDSU). Notes by Christina Hennessey

Agenda/Discussion

Review of what the committee accomplished this year (2021-2022)

Goals we originally set for 2021-2022

We had great accomplishments on cleaning up the wiki and getting the Analytics folders clean and reorganized. We did a review of GreenGlass vs. Ex Libris Analytics and the pros and cons for each. We continue to share recent free outside webinars and training opportunities on Analytics and Assessment on the wiki home page for all the CSU (and the world) to use. We kept up with CSU Analytics & Assessment questions in Slack, on the listserv, and through email. Also not listed on the goals: we created a survey to get information about the top issues for Analytics across the CSUs, followed up on those issues, and also to get interest in trainings & future webinars.

Plans going forward for the committee

Said goodbye to outgoing members Christina & Lisa; Kevin will be the new chair; Jung Ah & Shonn & Patrick are staying on, and the committee will have Natalya Serge (CSUSM) and Stacie Jensen (Sacramento State) on the group next year. The new group will elect a vice-chair and make annual goals for the group in August or September 2022.

Updates on the Top 3 CSU Analytics issues given to Ex Libris 

There has been no activity on the top 3 Analytics issues given to Ex Libris since our last meeting and since our follow-up with those that gave us more detail on those issues. Our reported issues were:

Analytics for Alma & Primo VE configuration changes and who changed them, retaining the whole history of each field change; Reporting from Primo Analytics too limited; Reporting on NZ-managed e-resources too limited

Kevin & Christina will meet with Ex Libris eventually along with other members of the team sometime in the coming months. 

Recent Analytics & Assessment webinars

Kevin shared his thoughts on the June 6, 2022 webinar on Overlap Analysis, and Christina shared some notes from a June 6, 2022 workshop on "Diving into Data using COUNTER & Tableau" from Northern Ohio Techinical Services Librarians. Please continue to share any outside webinars you find at regional conferences that we can post on the wiki.

Backing up Analytics reports

There was a helpful recent discussion (May 2022) on ALMA-L about backing up Analytics reports and protecting Analytics reports from deletion and overwrite through permissions. The committee will make a wiki page with this information and recommendations.

Humboldt name change in CSU+ reports/dashboards

The Resource Sharing group reported that some of the reports & dashboards needed to be updated to reflect the new name for Humboldt State University (Cal Poly Humboldt) as the data was now incorrect for that institution. Christina demonstrated how to update the bins in the reports to fix this problem. She found some of the them and fixed them but there might be more.

Next meeting: The group is dismissed for the summer. New Chair Kevin Phillips will reach out to the new 2022-2023 team in July 2022 to determine new monthly meeting times.

Discovery

Christian Ward  

Discovery Open forum - Rapido Edition

The DFC hosted our final open forum for the year, with a focus on discovery changes as a result of implementing Rapido. Topics discussed included:

Satisfaction Survey: Top-3 Issues

The DFC continues to work with Ex Libris on the discovery satisfaction survey top-3 issues. The issues presented to Ex Libris are:

1. uResolver matching issues

2. features not working: lc browse search, date newest filter, journal search, ranking, performance issues

3. inadequate analytics for assessing performance and usage of Primo VE

Ex Libris has provided some initial answers and acknowledges that some issues are being address via salesforce cases. 

Discovery and Rapido

Chris Lee provided an overview of the changes to the discovery UI/UX that will need to be configured to implement Rapido. Details of the changes are available online at Discovery and Rapido .

ERM

Ilda Cardenas (Unlicensed)

The ERM Committee did not meet this month. Jessica held a meeting regarding Alma Licensing Agreements in its stead.

Fulfillment

Kelly Ann Sam (Unlicensed)

In June the FFC reviewed the CSU Walk-In User Policy and updated it to be reviewed and sent out the listservs when the new committee begins. 

Resource Management

Brinna Pam Anan

The ULMS Resource Management Functional Committee met for the final time this year on Tuesday, June 7, 2022 from 1-2 PM.

1. Pam met with Jill on Friday, May 27 to do a preliminary handoff from Chair to Incoming Chair.  Issues discussed were Confluence updating, scheduling meetings, ULMS Governance structure, each Group/Task Force under the RMFC's purview.  Future handoff meeting with Brandon is scheduled for Monday, June 27.  New meetings to commence with Jill as Chair for 2022-2023.

2. The Top Three Resource Management issues and Ex Libris' comments were discussed.  Ex Libris offered information on working/advisory groups that are addressing the quality of metadata/records and CDI support (CZ Management Group, CDI Advisory Group, dedicated CDI Team).  One of the future goals of the RMFC is to gauge the level of participation and bandwidth the CSUs could have to participate in or contribute to these other working groups.

3. The future, structure, and goals of the RMFC were also discussed among the group along with committee members' experience of being a part of the RMFC and performing committee work during the pandemic.  Feedback will be sent to Brandon and then further discussed with the future makeup of the RMFC.

4. The Inclusive Description Task Force is working on an inclusive cataloging statement and accompanying reporting form.  The Task Force is also working on procedures to handle outdated American Indian subject headings.  The RMFC and Brandon have been informed and work will continue in the upcoming year.

Resource Sharing

Meghann Weldon (Unlicensed)

• The Assessment and Analytics committee will be reaching out to the Resource Sharing committee for collaboration on creating a Rapido Dashboard. Natalya Magazino will be sharing with the committee going the mechanics of the dashboard and how to create an analytics dashboard.  

• We will continue to monitor and update the ACRL IPEDS reports as needed.

• There may be opportunities to work more directly with SRDC on collection development analytics as well.

Discovery

Ian Chan (Unlicensed)

Evaluate Primo VE

We should evaluate Primo VE and assess whether it offers improvements over Primo Back Office. The CSU Libraries elected to migrate to VE because of Ex Libris described it as offering greater functionality over BO and solutions to problems found in BO.

Discussion

• There has been feedback describing performance issues with VE.

• How will we use the findings from the evaluation?

• When framing our questions, we should keep in mind that shortcomings or missing functionality might not be attributed to differences between VE and BO.

• Should survey have one broad question or address the major areas of Primo (e.g. user interface, search engine results, and other areas)?

• What issues carried over from BO to VE?

• A survey addressing Primo positives and negatives, separate from ULMS satisfaction survey, would be helpful in addressing issues with Ex Libris.

• The survey results would also be useful in comparing findings with peer institutions and coordinating discussions aimed at seeking improvements from Ex Libris.

Establish Test Sites

Establish 2-5 campuses that can monitor release notes and test new discovery features when available. This was proposed by Christian Ward to the ULMS Steering Committee.

Discussion

• The committee is supportive of this proposal and members indicated strong interest in participating.

• Christopher Novak, Hannah Lee , Julie Shen, and Keven Jeffery have offered to participate.

Update Wiki Content

Some pages in the Discovery area of the ULMS wiki require updates and possibly pruning.

Discussion

• One example is the Primo VE Known Issues - Not updated as of August 2022 page. Others might include pages found within the Primo VE Customization section.

Apply Findings from UI/UX Evaluation

Discuss findings from UX studies such as those conducted by Gabriel Gardner and Heather Cribbs, and provide a set of recommendations.

Discussion

• How does filter usage compare across campuses?

• How do CSU library websites direct users to Primo?

• How have campuses elected to include/exclude and expand/collapse facets?

• Should we consider A/B testing?

• Should we compile a list of Primo-related UX studies that have been completed, are under consideration, and in-process?

• Ex Libris conducts UX testing but does not have the capacity to conduct studies that would replicate the scope found within the CSU consortial and campus environments.

Address uResolver Matching Issues

Many consortia campuses are reporting records with uResolver matching issues. These include: 1) over matching of results, 2) under matching of results, 3) different format matching, and 4) title matching issues.

This priority is one of the three top issues presented to Ex Libris last Spring. Committee will address this at next meeting.

Address Features Not Working as Expected

Features are not working as expected or as we understand them to work. For example:

• LC subject browse does not return expected results,

• Date Newest facet only works when specific metadata is present in records, which is inconsistent in CDI and Alma records,

• Search within journal feature is nice, but only works when ISSN is present and can only be disabled using CSS,

• Ranking configuration still eludes many campuses in the consortia, and

• Performance issues with indexing, front-end load times, and feature activation.

This priority is one of the three top issues presented to Ex Libris last Spring.

It would be most helpful if campuses can describe the negative impact on their users.

Committee will address this at next meeting.

Investigate Improvements to Analytics Gathering and Analysis

This priority is one of the three top issues presented to Ex Libris last Spring. Committee will address this at next meeting.

ERM

Austin Lenzen (Unlicensed)

The ERM committee will be reaching out to the Discovery committee about the continued crossover between our two areas.  Given her background in Discovery, Christine Holmes could be a useful liaison.  Our ERM members have agreed that merged records should be our first point of conversation.

We have also begun identifying broken links and outdated information on the Wiki.  Quite a bit of our documentation has not been updated in the past two years.

Fulfillment

Rachael-Joy Davis (Unlicensed)

Goal 1: Finalize Walk-In User Policy

Overview:

Over the past few years, the Fulfillment Functional Committee (FFC) and Resource Sharing Functional Committee (RSFC) successfully convinced Ex Libris to make the necessary updates in Alma to make it possible for walk-in users to use CSU+. In the 2021/2022 FY, FFC’s goal was to write up a policy for approval by COLD that will ask all CSU libraries to use the new workflow and allow walk-in users at all campuses. Our goal for 2022/2023 FY is to finalize the policy language in collaboration with RSFC, submit the proposed policy to the Steering Committee for review/feedback, then hand it over to COLD for final review and approval of policy.

Assessment:

This goal is complete when a policy is submitted to COLD.

Goal 2: Alma/Primo Release Notes

Overview:

Keep the Access Services community aware of Alma updates by reading through Ex Libris release notes and testing new updates.

Assessment:

This is an ongoing goal. The goal is considered complete if the FFC disseminates information on Alma updates.

Goal 3: Satisfaction Survey

Overview:

Assist with dissemination of surveys and gathering and evaluating data from survey responses. FFC will identify areas of improvement with current functionality as it relates to Fulfillment, as well as what features are missing from the current product.

Assessment:

This is an ongoing goal. The goal is considered complete if the FFC gathers, compiles, and disseminates survey feedback to improve the user experience.

Goal 4: General Fulfillment/Access Services Survey

Overview:

Gain insight from Access Services staff as we work towards navigating post-pandemic library environments: structure, services, and user experience. Survey questions may include:

●      Discontinued services

●      New services

●      Service models

●      Staffing

●      Student employment

Assessment:

This goal is complete if survey results have been evaluated and disseminated across the CSU libraries, and a resource is provided for all libraries to gain insight into how other libraries are structured and how fulfillment service models differ across the 23 CSU libraries.

Resource Management

Jill Strykowski

• Look at Periodical records in Alma/Primo to see if there is anything we can do about the mismatched and duplicative records that appear when one searches for a Periodical by title in OneSearch.
  Jill will follow-up next meeting with recommendations for master or weighted records

• Recommend that we discuss the practice of putting what used to go in a 655 for electronic books into a local bib data field (Nerissa Lindsey, seconded by Jill)
  To look into for follow-up: Do we know how heavily facets are used? - do we use genre at all?

• Use of new secondary resource type feature - is this useful?

• Are we commenting on Metadoor? - getting involved in open access MARC record repository? minting record numbers for collocation - getting involved in Data excellence initiatives?

• Pulling people together to recommend greater cohesiveness in metadata between groups (ULMS, DAMS, IR’s, etc.)

Resource Sharing

Tina Dwyer

• Improving patron communication and marketing of Rapido and other interlibrary services

• Create more ways for resource sharing staff to report issues and possible improvements for RAPIDO

• Cleaning up and improving the WIKI

• RAPIDO content moved forward

• Archiving content that is no longer relevant 

• Reinvigorate task forces 

• Finalize walk in policy