20.05: Administration and Report Updates

by Kelly McElligott on Oct 14, 2020

New Features in Koha 20.05 for Administration and Reports

Details about new features coming in Koha 20.05 listed below.

New Feature : Plugin Architecture

[23975] Add ability to search and install plugins from GitHub

This patch makes public plugins discoverable from within Koha itself via a search box at the top of the plugin management page.
One can search for plugins and install them directly from their source.
WARNING: Plugins are not yet verified by the community, use at your own risk. The feature is disabled by default; to enable it an administrator must uncomment, or add new lines to, the relevant configuration lines inside the plugin_repos config block within koha-conf.xml.

[24302] Add a way to specify nested objects to embed in OpenAPI

This development introduces a way to define embeddable objects on API routes. On the path specs, they will be specified using dot notation for nested embeddings:
“x-koha-embed”: [
‘biblio’,
‘biblio.items’,
‘fund’
]
The consumer will need to add a header to the request, specifying the things they want to embed using comma-separated values like this:
x-koha-embed: ‘biblio,fund’
This header will be validated against the endpoint spec and an error code will be returned if the request is not appropriate.

New Feature: Web Services

[24369] Add ability to set CORS header in Koha

This development adds support for setting the Access-Control-Allow-Origin header in Koha using the new AccessControlAllowOrigin system preference. This is especially useful for integrating data from the services provided by Koha on sites other than Koha itself.
New system preference: AccessControlAllowOrigin defaults to empty.

New Enhancements: Architecture, internals, and plumbing

[18936] Move issuingrules into circulation_rules

This new feature is moving the data to a new table within Koha. This will not affect the staff interface's circulation rules. If a library has any reports that are built to query the circulation rules, these reports will need adjusting.

Architecture, internals, and plumbing

[19735] Move Perl deps definitions into a cpanfile
This enhancement moves us away from custom-built dependency management and to the widely adopted cpanfile format for perl dependency listing.
If you are running koha from git for development purposes you can now install perl dependencies using standard perl tooling and the included cpanfile.
This patch also introduces the ability to set maximum versions in our dependancy listing (and excluded versions too), which should help us better track our compatibility.
[20116] Improve performance by caching the language list
[20443] Move C4::Members::Attributes to Koha namespace
[20728] Remove subroutines GetLastOrder*
[21294] Add identification of boolean fields in the database
[21503] Update AuthorisedValues.pm to fall back to code if description doesn’t exist
[21746] Remove NO_LIBRARY_SET
[21800] TransformKohaToMarc should respect non-repeatability of item subfields
[22529] /svc/members/search relies on quirks of haspermission
[22589] Remove sub C4::Overdues::BorType
[23463] Move C4::Items CRUD subroutines to Koha::Item
[24052] Koha::XSLT housekeeping for bug 23290 (follow-up)
[24066] Koha::Patron->has_permission has no POD (POD plain old documentation)
[24356] objects.search prefetch
This enhancement makes the Koha::Object(s) derived classes expose information about prefetch-able relations. This is then used by a new helper to generate the prefetch information for the DBIC query.
[24368] Koha::Libraries->pickup_locations needs refactoring/ratifying
[24418] Add Koha::Biblio->suggestions
[24419] Add Koha::Suggestion->suggester accessor
[24430] Remove C4::Biblio::CountBiblioInOrders
[24435] Add Koha::Biblio->items_count
[24440] Add ->current_item_level_holds to Koha::Acquisition::Order
[24448] Add Koha::Biblio->subscriptions_count
[24455] Add ability to apply Koha formatting to dates from Javascript
This patchset lays the foundations for applying date formatting as described in the Koha system preferences to datetimes returned by the API in RFC3339 format.
[24463] Consistent accessor-relationship naming for basket_group in Basket.pm
[24467] *_count methods should be avoided
[24468] C4::Reserves::getitype is no longer used
[24529] Uninitialised value warnings in C4::Reserves
[24545] Replace Franklin Street by gnu.org/licenses in copyright
This enhancement updates the GNU GPL license and copyright statement in all files so they are the same. It also updates the QA check to catch all new files.
[24561] Add a datatables API wrapper
This patch adds a datatables wrapper that allows using datatables against Koha’s API. It implements:
- Server side pagination
- Filtering/searching
- Embedding related objects in the request
- Sorting and filtering by nested objects
[24642] Cache::Memcached::Fast::Safe must be marked as mandatory
[24715] Cache repeatable subfield in TransformKohaToMarc
[24721] emailLibrarianWhenHoldIsPlaced should use Koha::Library->inbound_email_address
[24723] EmailPurchaseSuggestions should use Koha::Library->inbound_email_address when set to ‘BranchEmailAddress’
[24726] overdue_notices.pl should use Koha::Library->inbound_email_address
[24732] Make DumpTemplateVars more readable
[24735] Remove QueryParser-related code
[24759] OpacRenewalBranch code should be a Koha::Item method
[24837] selectbranchprinter needs to be renamed
[24994] TableExists should be used instead of IF NOT EXISTS in updatedatabase

[25109] Add execution locking to Koha::Script
[25172] Koha::Logger init is failing silently
[25296] Add a way to force an empty Koha::Objects resultset
[25297] Consistent return value in K::A::Order->current_item_level_holds

[25303] Koha::Objects->delete should not skip overridden object class ->delete
[25070] Include files to display address and contact must be refactored
[21395] Make perlcritic happy

[22823] Koha::Library needs a method for obtaining the inbound email address

This patch adds a new inbound_email_address method to the Koha::Library class. This allows for a consistent way of getting a library's branch email address for incoming mail.

[24103] Add option to dump built search query to templates

This enhancement allows you to view the search query used by Zebra or Elasticsearch, to help with troubleshooting.

To use, enable the new system preference DumpSearchQueryTemplate, enable DumpTemplateVarsIntranet and DumpTemplateVarsOpac, and then search the page source in the staff interface or OPAC for ‘search_query’.

New system preference: DumpSearchQueryTemplate defaults to disabled.

Note: If this is needed, turn it on, but remember to turn it back to disabled after troubleshooting.

[25045] Add a way to restrict anonymous access to public routes (OpacPublic behaviour)

This enhancement allows libraries to distinctly disable the OPAC but allow the public facing API’s to be enabled.
New system preference: RESTPublicAnonymousRequests defaults to enabled.

If a library does not have an OPAC for their Koha, this must be useful.

New Feature: Authentication

[21190] Add logging of successful/unsuccessful login attempts
This enhancement adds two new logging preferences AuthFailureLog and AuthSuccessLog in order to keep track of bad login attempts and successful ones.
NOTE: In some countries, this may be a requirement as a local application of GDPR legislation.
New system preferences: AuthFailureLog and AuthSuccessLog both default to disabled.

New Features: Command-line Utilities

[15214] Add logging of authority updates to bulkmarcimport
[18414] Add ability to pass a file of borrowernumbers for deletion to delete_patrons.pl
Adds the ability to specify a file with the –file flag that
should be a list of borrowernumbers for deletion.
If used without other flags it will delete the list of borrowers, if used with other flags it will treat the other criteria as filters for the list.
[19008] Add more options to cleanup database script
The cleanup_database.pl cronjob now also includes options for deleting: entries from the statistics table,deleted bibliographic records and items (deletedbiblio, deletedbiblioitems, deletedbiblio_metadata, deleteditems) deleted patrons (deleted_patrons), returend checkouts (old_issues), filled and cancelled holds (old_reserves), finished transfers between libraries (branchtransfers)
[21177] Add ability to run misc/devel/update_dbic_class_files.pl without passing parameters by defaulting to koha-conf.xml
[21865] Add Elasticsearch support to, and improve verbose output of, remove_unused_authorities.pl
[23571] Add measures to prevent concurrent execution of fines.pl
Sponsored by Orex Digital
[23871] Add check for ‘title exists’ to search_for_data_inconsistencies.pl
[24340] Add ability to disable SIP using koha-sip
This enhancement adds –enable and –disable options to the koha-sip Debian package command.
Usage:
- koha-sip –enable instancename => Enables the Koha SIP server
- koha-sip –disable instancename => Disables and stops the Koha SIP server
[24526] Add verbose and test modes to the automatic_renewals.pl cronjob
This patchset adds new options to the automatic_renewals.pl script to allow test and verbose modes.
Important: The patches make the –confirm switch required, without it script will be run in test mode. Existing scheduled cronjobs will need to be updated to supply this switch.
Running without –confirm will default in verbose mode as well.
[24651] Add –maxdays option to the fines.pl cronjob to reduce the chance of re-processing very old, already capped, fines.
Improve the performances of the fines.pl cronjob by reducing the number of accountlines it targets by this new --maxdays option.
[24883] Add misc/load_yaml.pl utility script to allow manual loading of yaml data files
During the 20.05 cycle a number of improvements were made to the installation mechanisms to enhance the translation workflows for this area. As part of that work many existing translated .sql files were moved to a yaml based file.

[21591] Data inconsistencies – Item types and biblio level

Database

[18177] Remove unused columns in aqbooksellers
[22887] authorised_values is missing a unique constraint on category + authorised_value
[22987] Add biblioimages.timestamp

[25152] subscription.closed is a boolean and must be tinyint(1)

l18N/L10N

[13897] Use YAML files for installer data
[21156] Internationalization: plural forms, context, and more for JS files
[23790] fr-CA translation of ACCOUNT_DEBIT and ACCOUNT_CREDIT notices
[24063] Add Sami language characters to Zebra
This patch adds some additional characters to the default zebra mappings for Sami languages to aid in searching on systems with such data present.
[24211] Compress/uncompress translation files
[24262] Translate installer data in YAML format
[24583] Rewrite mandatory installation files to YAML
[24584] Rewrite optional installation files to YAML
[24593] Rewrite MARC21 optional data to YAML
[24594] Rewrite MARC21 mandatory data to YAML
[24648] Contextualization of past tense “Created”
[24662] Remove global variables MSG_* from datatables.inc

[24664] Add missing *-messages-js.po

Plugin Architecture

[24183] Introduce before_send_messages hook

This patch adds a new plugin hook to allow pre-processing of the message queue prior to sending messages.

Rest API

[18731] Add routes for acquisition orders
Sponsored by Camden County
This development adds API routes to perform CRUD operation on acquisition order lines.
[22615] Add routes for /ill_backends
[23893] Add ->new_from_api and ->set_from_api methods to Koha::Object
This development introduces generic methods to deal with API-to-DB attribute names translations, and some data transformations (dates and booleans).
With this design we can overload this methods to handle specific cases without repeating the code as we did on initial implementations of API controllers.
Testing becomes easier as well.
[24228] Add a parameter to recursively embed objects in Koha::Object(s)->to_api
This patch introduces a parameter to the Koha::Object class (’embed’) that should be a hashref pointing to a data structure following what’s documented in the code. This parameter allows the caller to specify things to embed recursively in the API representation of the object. For example: you could request a biblio object with its items attached, like this:
$biblio_json = $biblio->to_api({ embed => { items => {} } });
The names specified for embedding, are used as attribute names on the resulting JSON object, and are expected to be class accessors.
The main use of this is the API, as introduced by bug 24302.
Koha::Objects->to_api is adjusted to pass its parameters down to the Koha::Object.
[24321] Make objects.search use mappings from Koha::Object(s)
This development takes advantage of the mappings that have been added to the Koha::Object level, and refactors the objects.search Mojolicious helper so it uses it internally.
This allows us to remove the ‘to_model’ parameter, and makes the need of any kind of mapping on the controllers irrelevant. All the existing mappings are removed and the controllers simplified in this move.
[24461] Add to_api_mapping to Koha::Acquisition::BasketGroup
[24464] Add K::A::Basket->creator
[24502] Add a query language and param (q=) to the API
[24528] Add a syntax for specifying counts on x-koha-embed
Sponsored by ByWater Solutions
[24615] Add support for ordering by related object columns in the REST API
[24700] Improve Mojo startup speed for REST APIs
Sponsored by National Library of Finland
[24908] Allow fetching text-formatted MARC data
[24909] Add unprivileged route to get a bibliographic record
[25032] Generic unhandled exception handling
The current code in the controllers is a bit heterogeneous regarding how unhandled exceptions are treated.
This enhancement introduces a generic way to write ‘something happened’ as a fallback after expected exceptions handling. This way the catch blocks are easier to read, and devs can follow this simple pattern when writing their endpoints.
[25279] Make the cities list use the API

SIP2

[15253] Add Koha::Logger based logging for SIP2

[20816] Add ability to send custom field(s) containing patron information in SIP patron responses

System Administration

[4944] Create separate noItemTypeImages preference for OPAC and Staff Client

With this patch, the noItemTypeImages preference will be used for the staff interface, while a new preference OpacNoItemTypeImages is added for the OPAC. For existing installations, the OpacnoItemTypeImages will be set to the same value as noItemTypeImages on the update, so there is no change in behavior.
New system preference: OpacNoItemTypeImages defaults to disabled.

[5614] Add sections/headings to Patron system preferences tab

This enhancement organizes the patron system preferences into sections. This makes them easier to find, instead of being one long unorganized list. Now system preferences within the Patron section will be organized into sections such as Patron Forms, Membership Expiry, Notices, and Notifications, Privacy and more!

[15668] Add column configuration to the items table in staff detail pages

Under the column configuration, there is a tab for Catalog. Here the table is called, holdings_table which can determine what is showing under the full record of the record. A library can choose to hide homebranch, holdingbranch, itype, collection code and more!

[15686] Rename “item level holds” circ rule column to “OPAC item level holds”

The column, Item Level Holds, in the circulation and fine rule table can be confusing to users of Koha. The column has now been renamed to be more specific on the definition of what this controlled, "OPAC item level holds".

[20399] Remove “did you mean” for the staff interface

This option didn't have any functionality to it, so this will be removed until it has options for the staff interface.

[20415] Remove UseKohaPlugins system preference

The system preference, UseKohaPlugins, is getting removed. The option to use plugins will always be on, a library will just need to upload one without having to turn on a system preference also.

[20484] Always show Elasticsearch configuration page when permission is set

If user has permissions, this Elastic search page option will be present. The permission is called, "manage_search_engine_config" found under Admin. If a user does not have permission, they will not see this Elastic search page.

[20648] “Display in check-out” renamed to “Display in patron’s brief information” on patron attributes configuration page

This new feature change goes along with this bug [12159], Koha will start displaying patron attribute information on the side panel above the tabs, if the patron attribute has been noted to show there. Koha is changing the wording in this feature to be more clear for users of Koha.

[21520] More complex OAI sets mappings

Prior to this patchset, the rules used to create OAI sets are processed with the ‘or’ boolean operator between each rule. This patch allows to use ‘or’ or ‘and’ between the rules. The evaluation of the rules is done according to the boolean operatorsprecedence: AND has a higher precedence than OR.

[24193] Add CodeMirror linting of JavaScript, CSS, HTML, and YAML system preferences

This enhancement adds CodeMirror plugins for linting system preferences that include JS, CSS, HTML, and YAML. When invalid data is entered in a linted CodeMirror editor an icon is displayed in the editor’s “gutter.” Hovering over the icon displays the error message.

[24291] Explanation next to limit item types by library is confusing

This enhancement updates the explanation on the item type add and edit form for the ‘Library limitation’ field. The text now says “Select ‘All libraries’ if all libraries use this item type. Otherwise, select the specific libraries that use this item type.”

[24475] Reword FinesMode system preference

A cleared definition of this system preference, FinesMode, has been added to Koha 20.05. For new users of Koha, clearer representation of the full use of this system preference is needed. Now in the system preference, FinesMode, there is a more clearly defined definition :

Prior to 20.05, the system preference read like this.

NOTE: Fines can also be charged by the CalculateFinesOnReturn system preference.

Koha 20.05 definition of system preference:

NOTE: If the cronjobs/fines.pl cronjob is being run, accruing and final fines will be calculated when the cron runs, and accruing fines will be finalized when an item is returned. If CalculateFinesOnReturn is enabled, final fines will be calculated when an item is returned.

[24576] StoreLastBorrower preference is in the wrong tab and is confusing

Prior to this feature change, the system preference, StoreLastBorrower, was under the OPACPrivacy. Now in Koha 20.05, the system preference will be located under the Circulation tab -Checkin area. This makes much more sense and will be easier to find.

[24844] Focus on the system preferences searchbar when going to admin home

This allows for ease of use going into the Administration module. Now when a user clicks Admin, the cursor automatically will go into the search System Preference field box.

New Features in Templates

[10469] Display more when editing subfields in frameworks

This new feature will change editing frameworks for the better! While editing the subfields in frameworks Koha will expand the entire editing area for less clicking.

[15352] Use URLLinkText instead of URL for item links

Adding a URL in the 952$u has been improved. Now a library can have specific text to be inserted instead of seeing the URL text. If a library would like a specific phrase to appear in the $u, this can be filled out in the new system preference, URLLinkText. By default, the wording will display as "Link to Resource" if a value is not filled into the system preference.

[23268] “Suspend all holds” calendar allows to select past date

The date picker will now not allow a date to be chosen in the past. This will apply to the suspend all holds feature.

[22468] Standardize on labeling ccode table columns as collection

Obtaining consistency throughout Koha in the wording around Collection Code. Koha now displays the descriptions everywhere, so areas, where the code was used before, will now show descriptions. This includes areas:

- The relative's checkouts table when displayed for the guarantor

- The new order and order receive forms in acquisitions (item creation)

- The description of CCODE in the authorized values table

- The column description on results in item search

- The branch transfers page

- Course reserves: add reserve items (one and in batch)

- Patron account > statistics tab

Other Template Changes

[16457] Remove the use of “onclick” from the patron entry form
[23533] Reindent patron entry form (memberentrygen.tt)
[23534] Use patron-title.inc on patron entry page
[23856] Split author and ISBN/ISSN out of citation in staged MARC record management
[23884] Merge strings.inc and browser-strings.inc
[23889] Improve style of menu header in advanced cataloging editor
This enhancement updates the styling of dropdown menu headers to make them apply more consistently across the system.
[24181] Improve the display of our datepickers
[24341] Add support for more complex markup in OPAC confirmation dialogs
[25135] Improve clarity and navigation of columns settings administration
[25416] Add information about anonymous session for XSLT use
Sponsored by Universidad ORT Uruguay
[25363] Merge common.js with staff-global.js
[25593] Terminology: Fix “There is no order for this biblio.” on catalog detail page
[25627] Move OPAC problem reports from administration to tools
[25687] Switch Y/N in EDI accounts table for Yes and No for better translatability
[25471] Add DataTables to MARC subfield structure admin page for bibliographic frameworks
[25747] Don’t display a comma when patron has no firstname

Critical Bugs Fixed

Although this is not the complete list of critical bugs fixed in this section, here are a few:

Architecture, internals, and plumbing

[13193] Make Memcached usage fork safe
Sponsored by National Library of Finland
Important Note: You will need to make sure you install Cache::Memcached::Fast::Safe to continue to use memcached after this.
[21674] Data integrity not enforced for library groups
[21761] Koha::Object supports passing through ‘update’ which means we can side step ‘set’ + ‘store’
[22522] API authentication breaks with updated Mojolicious version
[23185] Koha::Objects supports passing through ‘update’ which means we can side step ‘set’ + ‘store’
[23290] XSLT system preferences allow administrators to exploit XML and XSLT vulnerabilities
This patchset refines the XSLT processing configuration such that we are more secure by disallowing the processing of external stylesheets by default and adding a configuration option to re-enable the functionality.
[24243] Bad characters in MARC cause internal server error when searching catalog
[24263] borrowers.relationship should not contain an empty string
[24552] Koha does not work with Search::Elasticsearch 6.00
[24719] C4::Context::set_remote_address() prevents file upload for non-Plack Koha
[24727] Typo in circulation.js
[24741] Recent creation of unique index on library_groups erroneously removes rows
[24754] UserEnv not set for ISLDI requests
[24788] Koha::Object->store calls column names as methods, relying on AUTOLOAD, with possibly surprising results
[25009] opac-showmarc.pl allows fetching data directly from import batches
[25040] Problematic current_timestamp syntax generated by DBIx::Class::Schema::Loader
[25131] Web installer broken if enable_plugin is set
[25142] Staff can access patrons’ infos from outside of their group
[25481] koha-plack not working under D10
[25485] TinyMCE broken in Debian package installs
[25567] borrower_attribute_types.category_code must be set to undef if not set
[25608] (regression) Inventory is broken
[25634] koha-foreach exits too early if any command has non-zero status
[25707] Mappings update in bug 11529 causes incorrect MARC to DB data flow
[24986] Maximum row size reached soon for borrowers and deletedborrowers

Database

[8132] Batch delete tool deletes items with holds on them
[13518] Table borrower_modifications is missing FK and not deleted with the patron
[24377] Record branch in statistics for auto-renewal

REST API

[24191] Sorting doesn’t use to_model
Sponsored by ByWater Solutions
[24432] order_by broken for date columns
[24487] build_query_params helper builds path parameter with matching criteria
[25411] Plugin routes cannot have anonymous access
Sponsored by ByWater Solutions
[24003] REST API should set C4::Context->userenv
[25774] REST API searches don’t handle correctly utf8 characters

Reports

[25000] SQL report not updated

SIP2

[23403] SIP2 lends to wrong patron if cardnumber is missing
[23640] L1 cache too long in SIP Server
[24175] Cannot cancel holds – wrong parameter passed for itemnumber
[24800] Koha does incomplete checkin when no return date is provided
[24966] Fix calls to maybe_add where method call does not return a value

System Administration

[24329] Patron cardnumber change times are lost during upgrade for bug 3820
[24670] Circulation and fine rules page has performance issues since issuingrules change
[25601] Error when unsetting default checkout, hold and return policy for a specific library
[25617] Error on about page when TimeFormat preference is set to 12hr
[25651] Modifying an authorised value make it disappear

Templates

[24241] Description missing for subpermission manage_accounts
[24713] JavaScript error on staff client catalog search results page
This patch modifies the output of template toolkit variables so that values in the in-page JavaScript are quoted. This avoids JavaScript errors when the template variable is empty.
[25839] Typo patron.streetype in member-main-address-style.inc