What’s of interest? Dealing With Bots: A COAR Resource for Repository Managers: Home
Tell me more!
Over the past few years, repositories have been encountering a growing number of bots trying to access their resources. These bots, or crawlers, navigate the internet, gathering data and indexing information for search engines and large language models, as well as for other purposes. While some of these bots are rather innocuous, others are sufficiently aggressive that they cause service disruptions in repositories (and other scholarly communications infrastructures).
Machine users have always been a critical constituency for repositories as search engines, aggregators, and other indexing services represent the predominant mechanisms whereby a repository’s resources are discovered. As such, it is very much in the community’s interest to ensure that repositories remain open and accessible to friendly bots and crawlers despite the increasingly aggressive nature of some bots.
In early 2025, the Confederation of Open Access Repositories (COAR) conducted a survey in early 2025 to assess the extent to which repositories were being adversely affected by a sudden and large increase in activity by “bots” - automated Web clients such as crawling or harvesting systems. The ensuing report - The impact of AI bots and crawlers on open repositories: Results of a COAR survey, April 2025 - was published in June 2025.
Where is it?: https://dealing-with-bots.coar-repositories.org/
This is one among many items I will regularly tag in Pinboard as oegconnect, and automatically post tagged as #OEGConnect to Mastodon. Do you know of something else we should share like this? Just reply below and we will check it out.
Or share it directly to the OEG Connect Sharing Zone