Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Funnelling locusts – further reflections on the OAPEN Library and DOAB’s response time 

In my previous post on our less than optimal performance of the OAPEN Library and DOAB, I wrote about the effect of AI bots on our systems, and that it has become a common problem for many sites providing open access (OA) or open source software content. All of them – including the OAPEN Library and DOAB – see a new threat: not censorship or budget cuts, but bots. A new kind of digital swarm has arrived, and it’s putting serious pressure on our infrastructure, especially the OAPEN Library. In this blog, I will discuss in some detail our ongoing work to prevent us from being overrun by AI bots. 

Interacting with OAPEN and DOAB – a dialogue 

At its core, every interaction with the OAPEN Library is a dialogue between a user and our server. A request is made, we respond. This simple back-and-forth happens countless times a day. A user is either a person, or an automated system: a bot.  

Most of the time, these interactions are respectful. Many bots—like those from Google or CLOCKSS—identify themselves and behave in a predictable way. We welcome not just human users, but also automated systems – for instance for text and data mining – into the OAPEN Library; we aim to share a quality-controlled collection of open access books as widely as possible.  

But with the rise of AI tools—especially large language models (LLMs)—the nature of these conversations is changing. Many bots no longer introduce themselves. They mask their identities and arrive in droves, flooding the system with requests. 

OAPEN under attack: a swarm of locusts 

Under normal circumstances, we see less than 50 dialogues per second happening between a user and the OAPEN Library. However, when a swarm of AI bots starts hammering the OAPEN server, the number of interactions rises to over 10,000. As you can imagine, this is similar to undergoing a swarm of locusts. 

These swarms of AI bots are scraping content to feed LLMs. The impact? Our response time balloons from milliseconds to minutes, effectively making the OAPEN Library unusable. And it doesn’t stop. These surges can last for hours and sometimes occur multiple times a day. 

Figure 1: Line drawing of locusts, taken from https://commons.wikimedia.org/wiki/File:Diagrams_of_Locusts_which_swarmed_over_England_in_1748.jpg  

Our solutions to the problem 

Stopping this digital onslaught is far from simple. The attacks don’t come from a single source or follow an obvious pattern. As an open infrastructure serving a global audience, we cannot block entire countries, as some may have done as a solution. Our goal is to make sure that all genuine users of the OAPEN Library continue to have access, and to funnel the flood of AI bots. Together with our colleagues of the CERN Data Center we have been working on several countermeasures. 

What measures have we implemented? 

  1. Blocking known offenders 

We keep a list of bots that have repeatedly overwhelmed the system and block them. While many bots disguise themselves, some of them are still recognisable. 

  1. Rate limiting 

When a user is sending more than a certain number of requests per seconds, we temporary block them. Our OAPEN Library server does not respond with an answer or by sending a file but instead gives the message “429 Too Many Requests”. 

We are applying the following limitations to handle these instances: 

  • More than 40 requests in 10 seconds for a URL starting with /discover or /handle will lead to a time-out. 
  • More than 20 requests in 10 seconds for a URL starting with /bitstream will lead to a time-out.  

If this results in genuine users being blocked from using the OAPEN Library, such as large institutions, we will recosider changing the limits. 

  1. Separating file downloads from information requests 

The most important function of the OAPEN Library is providing access to OA book and chapter files. To ensure that they remain accessible, requests for files are now redirected to four separate servers to manage file downloads. This helps us deal with the large number of file requests, as there are now more servers to carry the load. 

  1. Continuous monitoring 

Additional to all the above measures, in collaboration with our colleagues from the CERN Data Center, we will continue to closely watch traffic to the OAPEN Library and adjust our responses as needed. 

Figure 2 An overview of response times of several aspects of the OAPEN Library 

Looking Ahead 

There is no silver bullet. Combating this problem requires significant time, technical resources, and coordination between our team and the CERN Data Center. On the horizon, a newer version of DSpace promises greater flexibility and tools to help manage these issues more effectively. Our current version of DSpace uses a relatively old version of the SOLR search engine and it is not possible to add an additional copy of the search engine to double the capacity. The latest versions of DSpace are using an updated version of SOLR, making it possible to add more capacity.  

Until then, we continue to do all we can to keep the OAPEN Library open and accessible – for humans and well-behaved bots alike – to avoid sincere users being harmed by these solutions. However, if you run into issues, please contact us

OAPEN and Jisc collaborate on enhancing the open policy finder

Open access (OA) policy information is increasing, as more funders, publishers, and institutions continue to develop OA book specific policies. Two resources aggregating OA book policy information include the PALOMERA Knowledge Base and the Jisc open policy finder (formerly Sherpa services). The Knowledge Base includes policies focused on funder and institutional policies, whereas the open policy finder focuses mostly on publisher policies.   

For many years, OAPEN has hosted a ‘Funder compliant publishers’ webpage to help researchers identify publishers who are aligned with the OA requirements from funders, such as the European Research Council (ERC), Wellcome, the Swiss National Science Foundation (SNSF), and the Austrian Science Fund (FWF).  

As Jisc is already collecting publisher policies, we decided to collaborate more closely with them on the open policy finder and share these policies we’ve collected over the years, to avoid duplication of efforts. As a result, we will be discontinuing the ‘Funder compliant publishers’ webpage from Friday 30 May

The open policy finder is an up-to-date resource for OA publisher policy information. This change reflects our shared commitment to improving the discoverability of OA book policies for the research community. The PALOMERA Knowledge Base continues to be the most comprehensive resource for funders and institutional policy OA books in the European Research Area where new policies are added on an ongoing basis.  

What will happen to the list of publishers on this webpage? 

All relevant policies from the original OAPEN webpage have already been added to the open policy finder and are discoverable in it. We have already contacted these publishers to inform them of this change.  

What should publishers do if they have a compliance policy for an OA funder? 

The open policy finder reflects policies as shared on publisher websites. 

  • Check your listing on the open policy finder to verify accuracy. 
  • If updates are needed, contact help@jisc.ac.uk with “open policy finder books” in the subject line. 
  • If your policy has not yet appeared, you can request its inclusion. 

We encourage all publishers with books listed in the OAPEN Library to ensure their open access policies are visible in the open policy finder. Doing so supports greater transparency and helps authors and institutions make informed decisions about publishing in open access. 

The open policy finder team also encourages you to join the publishers jiscmail group (sherpapublishers@jiscmail.ac.uk) to keep abreast of service updates. 

Are there other resources for OA book polices? 

The PALOMERA project, and 24-month EU-funded project looking at policy alignment for OA book policies in the European Research Area (ERA), delivered a list of recommendations for publishers to promote transparency and high standards in OA publishing. Recommendations are also available for other stakeholders, such as libraries, research performing organisations, research funding organisations, and more.  

The OAPEN OA Books Toolkit is another useful resource and as part of the PALOMERA project, was updated to include sections on OA book policies and OA book funding policies

About Jisc

Jisc is the UK digital, data and technology agency focused on tertiary education, research and innovation. They are a not-for-profit organisation and believe education and research improves lives and that technology improves education and research. They provide managed and brokered products and services, enhanced with expertise and intelligence to provide sector leadership and enable digital transformation. 

Launching our UKRI collection

On 6 December 2024, we announced that we entered a partnership collaboration with UK Research and Innovation (UKRI) to provide a set of dedicated services for its funded open access (OA) books. 

Under the terms of this partnership collaboration, it was agreed that all OA books supported by UKRI are made available as a funded collection in the OAPEN Library from January 2025, alongside our other funded collections (including, DFG, FWF, NWO, SNSF, Wellcome, SCOAP3, Knowledge Unlatched, and the ERC). 

In the six months that have passed since this announcement, we’re pleased to report that 46 books and 5 chapters have already been added to the UKRI collection out of the 90 OA books they have supported so far. We are regularly adding to the collection as more titles are published and supported.

We also launched a dedicated webpage showcasing the UKRI collection. The webpage provides more information for publishers, researchers, and libraries, and on the UKRI open access policy

Other services included in the partnership collaboration are archiving of funded OA books in PORTICO, distribution to libraries globally through our metadata feeds and aggregators, reporting on COUNTER usage in the OAPEN library through a personalised dashboard, and additional discoverability through seamless integration of funded OA books in the Directory of Open Access Books (DOAB) through the DOAB Trusted Platform Network

We look forward to continuing this partnership collaboration with UKRI and working together to achieve an open future for academic monographs. 

Browse the UKRI collection in the OAPEN Library. 


About UKRI

UK Research and Innovation (UKRI) is the largest public funder of research in the UK. UKRI brings together the seven disciplinary research councils, Research England, which is responsible for supporting research and knowledge exchange at higher education institutions in England, and the UK’s innovation agency, Innovate UK. UKRI supports research across all disciplines through a large variety of grants, programmes, and investments. 

Two Years On: A Conversation with DFG on Supporting Open Access Books


It has been two years since OAPEN launched a dedicated collection of DFG-funded open access books. Part of this collection is supported by the DFG Open Access Publication Funding programme. To mark this occasion, OAPEN’s Funder Relations Manager, Laura Bandura-Morgan, interviewed Michael Geuenich, Programme Officer, Scientific Library Services and Information Systems, and Dr. Angela Holzer, Programme Director. Scientific Library Services and Information Systems, of the Deutsche Foreschungsgemeinschaft (DFG). 

Thank you for accepting our invitation for this interview.

1. To begin with, could you tell us a bit more about this funding programme? 

The “Open Access Publication Funding” programme is designed to grant a fixed allowance for the publication of research results via open access. Only research institutions, not individuals, may apply for this allowance – the institutions in turn can use the allowance to endow their Open Access publication funds to enable their researchers to publish their research results. This includes the publication of journal articles as well as open access books (monographs, edited volumes).

This funding programme highlights that open access plays a pivotal role in improving scientific communication and that its costs should be seen in relation to this aim. The aim of this funding programme is to create structural adjustment in financial flows, thus enabling the transformation of open access and improving transparency regarding the costs for open access publication of research results.

The funding of open access publication costs thus is firstly directed towards those research institutions that require it due to the transformation towards publication-based accounting. The provision of funds should secondly result in the formation and development of adequate structures at the institutions receiving funding, to enable them as far as possible to identify the costs of publications in an automated and standardised manner. Therefore, the programme aims to support institutions with the introduction and development of such structures by subsidising researchers’ fees for open access publications.

Open access transformation ultimately requires structural adaptations to the entire system for financing the costs of scholarly publishing. The third aim of the programme is therefore to create a transition to open access funding in which research funders assume financial responsibility for the publications arising from the research they fund.For all OA books funded under the funding programme, quality assurance is ensured by the requirement that the funded books meet the quality standards of AG Universitätsverlage (Association of University Presses in Germany, Austria, Switzerland and South Tyrol, https://ag-univerlage.de/?page_id=2563).

2. What has been DFG’s experience with it so far, and does it meet the expectations of the researchers it is designed to support?

Since the start of the programme in 2021, the DFG has approved funding for open access publications with a volume of 109.3 million euros. The success of the programme can be seen not least in the fact that the programme was ‘overbooked’ in all application years, i.e. a large number of successful proposals were approved, meaning that across-the-board cuts had to be made to the approved funds per publication.

Approximately 4% (4.5 million euros) of the 109.3 million have been approved for the funding of Open Access monographs resulting from DFG projects.

We and the funded libraries have observed that the funding has definitely resulted in a stronger boost in the area of monographs and has created more awareness also among researchers in the Humanities for the importance of Open Access.

We are also noticing an interesting development in the publication of books. The number of books actually funded remains still below the number applied for. One of the possible reasons for this is that the specific publication processes for books follow a different timescale. The publication of a monograph or an edited volume requires more time than the publication of journal articles, and therefore the development takes longer and the funds cannot be used immediately. 

3. Why did you choose to partner with OAPEN to host your funded OA books?

With regard to the books funded through the Open Access Publication Funding Programme, it seemed important to us to have a central place, a single point of contact for all books resulting from this funding. OAPEN is a reliable and community-oriented partner that has been doing valuable work for many years, and which itself emerged from a EU co-funded project. The close links with the Directory of Open Access Books (DOAB) also ensure the visibility of the funded open access monographs.

The current situation in science politics shows that ‘lots of copies keep stuff safe’; the decision to also deposit the monographs in the OAPEN Library as a repository therefore proved to be the right one. In addition, we have also included in the collection all publications that have been submitted by publishers cooperating with OAPEN and bear a reference to the DFG in the funding acknowledgement. This provides a valuable overview of a selection of the Open Access monographs funded by the DFG via research projects.

Of course, not all Open Access monographs resulting from DFG funding or DFG-funded projects are included in the collection yet. Those publications that were not funded through the “Open Access Publication Funding” programme or that have not been published by one of the publishers cooperating with OAPEN are not included. We would therefore like to see more scientific publishers seeking cooperation with OAPEN so that the OAPEN quality standards can be guaranteed for these publishers.

Since the launch of this programme in 2021, DFG has initiated several other initiatives related to funding open access books. One recent example is the National Service Point for Diamond Open Access (SeDOA), which is set to officially begin on May 1st.

4. Could you elaborate on the key objectives of SeDOA, especially those that are related to OA books?
(e.g. What initial steps will be taken to realize the project’s goals? How will the project support Diamond Open Access book publishing?)

SeDOA is the consortium of 15 institutions that won the ‘Fresh Momentum for Open Access’ call for proposals, which the DFG organised in 2024. The objective of this call was to boost the performance capacity of Diamond Open Access infrastructures located at publicly organised institutions in Germany or operated by non-commercial and non-profit actors in Germany.

SeDOA will act as a central hub for Diamond Open Access services – it will match journals, book series or individual publication projects with appropriate Diamond Open Access publishing services.

The work of SeDOA will not only focus on journals, but will also explicitly look at books. The participating institutions will contribute their expertise in scholar-led open access publishing for books. The consortium includes the University Library of Heidelberg with its Heidelberg University Publishing, the Technical University of Berlin with its university press (as part of Berlin Universities Publishing) and ZB MED in Cologne. Further institutions associated through AG Universitätsverlage contribute the expertise of many years of experience in support and development of services for OA Books. Furthermore, Bielefeld University Library was involved in the EU Horizon Europe project “PALOMERA” that sought to to accelerate the open access transformation of books by aligning OA book policies.

Besides other tasks SeDOA will offer workflow-related basic services like layout, writing environments, editorial software, hosting etc. and will also provide support for transitioning publications into the non-commercial open sector.

5. Do you anticipate that SeDOA will lead to any changes or adaptations in the existing Infrastructures for Scholarly Publishing programme—particularly in terms of financial support for open source publishing platforms, digital preservation services, or infrastructure development for books?

With the approval for SeDOA, the existing funding programme “Infrastructures for Scholarly Publishing” was also adapted. Since the end of last year, applicants have been required to first use the services of SeDOA if, for example, they are planning to establish a new Diamond Open Access publication.

Because SeDOA offers workflow-related basic services, it is now possible for editors of Diamond OA journals not to have to submit a specific funding application (with an uncertain outcome) for many works, but to use the services of SeDOA instead.

We expect that the balance will change for those applications in the funding programme that relate to the publication of papers: Basic services will be covered by SeDOA, but the further development of platforms, tools and business models will move more into the focus of funding.

At best, SeDOA and the projects funded in “Infrastructures for Scholarly Publishing” will complement each other, so that SeDOA can also help to disseminate the knowledge and progress gained in the projects to the entire Diamond Open Access community in Germany.

Lastly, how do you see funding for OA books developing in the future?

Open access books have a different speed of open access transition than journal articles or conference papers. This can be observed particularly in scientific disciplines such as the humanities and social sciences, which still publish monographs more than other disciplines. For subject areas like art history or architecture, for example, there are challenges that other areas do not face in the same way – for example, the (often impossible) open access licensing of image material. The removal of such barriers could potentially be a topic for the Policy Forum for Open Access Books that will take place the first time in May.