Best Practices for Pipe Configuration and Publishing Schedules

Background

Because the ingestion of updates and new data will have an effect on all institutional as well as the consortial Primo views, the procedure and schedule through which current and new data pipes are run needs to be transparent.  In addition, campuses need a procedure to get new local data sources into the consortial Primo knowledgebase.

Policy Statement

This policy describes best practices for Primo publishing pipe configurations and schedules. It may address how changes are submitted, determined and implemented.

Best practice recommendations

Note: This workflow must be coordinated with the Primo Normalization Rules workflow.

Pipes Management

1. A CSU Consortium Primo Publishing Manager (PPM) shall be appointed as part of the ULMS Program Management Team.  The PPM shall be responsible for setting up and monitoring pipes that publish records to Primo.  A PPM backup will also be appointed.

2. The PPM will apply Normalization Mapping Sets (templates) for each pipe based on norm rules approved through the Primo Norm Rules/Search Engine Working Group process.

3. Requests from new data source pipes will be submitted by the campus ULMS manager or groups within the consortium directly to the PPM, who will evaluate their requirements and impact on the system.

4. Pipes for new data sources will be configured using an existing NR template if available.

5. If necessary, the CSU Consortium shall establish specialized working groups to recommend NR templates for new data sources such as digital collections, finding aids, open educational resources, and other types of digital objects.

6. Proposed new data source/pipes and pipe schedule changes will be referred to the the ULMS Program Management Team for approval.

7. A monthly report on new pipes will be published to the consortium.  The report will contain information such as

   1. what PNX template will be used for the pipe

   2. if there’s no extant template, what work is being done to create one

   3. when the pipe will be tested

   4. when the pipe will move to production

8. The initial ingest of a data source will be scheduled so as to have the lowest impact possible on resource availability during peak use times.

9. Recommended changes and additions to NR templates for new data sources will be subject to the Primo Norm Rules/Search Engine Working Group process.

Pipes Scheduling

1. Pipes will be scheduled so as to have the lowest impact possible on resource availability during peak use times.  

2. The PPM will maintain a public schedule of pipes and report on the results, run time, and success/failure of each pipe run (example: (document only goes through April 2018)

3. The scheduling will be subject to consensus across the consortium, and may change depending on how many pipes need to run, and how often each pipe needs to run.  

4. The following sample schedule is initially recommended to run seven days a week. (source: Orbis Cascade documenation, link no longer available)

•  

  • Primo publishing pipe starts at 4:30 AM

  • Primo dedupe pipe starts at 5:30 AM

  • Indexing and Hotswapping job starts at 6:00 AM

  • Updating normally occurs around 9:00 AM

•  

  • Primo publishing pipe starts at 7:30 PM

  • Primo dedupe pipe starts at 8:30 PM

  • Indexing and Hotswapping job starts at 9:00 PM

  • Updating normally occurs around 1:00 AM

Procedures in Primo

See ExLibris documentation for setting up and scheduling pipes.

Action log

Tasks to be completed