[Bug]: The OPcache interned strings buffer is nearly full

⚠️ This issue respects the following points: ⚠️

This is a bug, not a question or a configuration/webserver/proxy issue.
This issue is not already reported on Github (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
I agree to follow Nextcloud's Code of Conduct.

Bug description

The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 8.

Steps to reproduce

Server Overview

Expected behavior

NA

Installation method

Manual installation

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Default user-backend (database)
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "1.1.1.14",
            "cloud.next.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "23.0.2.1",
        "overwrite.cli.url": "https:\/\/cloud.next.com",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "simpleSignUpLink.shown": false,
        "trashbin_retention_obligation": "auto,30",
        "versions_retention_obligation": "auto,1",
        "skeletondirectory": "",
        "default_phone_region": "US",
        "allow_local_remote_servers": true,
        "mail_smtpmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_smtpport": "587",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "maintenance": false,
        "updater.release.channel": "stable",
        "theme": "",
        "loglevel": 2,
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - admin_audit: 1.13.0
  - bruteforcesettings: 2.3.0
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contactsinteraction: 1.4.0
  - dashboard: 7.3.0
  - dav: 1.21.0
  - federatedfilesharing: 1.13.0
  - federation: 1.13.0
  - files: 1.18.0
  - files_external: 1.15.0
  - files_pdfviewer: 2.4.0
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - firstrunwizard: 2.12.0
  - groupfolders: 11.1.2
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - nextcloud_announcements: 1.12.0
  - notifications: 2.11.1
  - oauth2: 1.11.0
  - officeonline: 1.1.2
  - password_policy: 1.13.0
  - photos: 1.5.0
  - previewgenerator: 4.0.0
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - ransomware_protection: 1.12.0
  - recommendations: 1.2.0
  - sendent: 1.2.10
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - socialsharing_email: 2.4.0
  - spreed: 13.0.3
  - support: 1.6.0
  - survey_client: 1.11.0
  - suspicious_login: 4.1.0
  - systemtags: 1.13.0
  - text: 3.4.0
  - theming: 1.14.0
  - twofactor_admin: 3.2.0
  - twofactor_backupcodes: 1.12.0
  - twofactor_totp: 6.2.0
  - updatenotification: 1.13.0
  - user_status: 1.3.1
  - viewer: 1.7.0
  - weather_status: 1.3.0
  - workflowengine: 2.5.0
Disabled:
  - circles: 22.1.1
  - encryption
  - user_ldap

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

.

@MichaIng

cc @MichaIng

I did a sudo service apache2 restart and it went away but then comes back after a while

@AndyXheli

@AndyXheli
Why do you think this is a bug, or was it just the best issue template you found? This shows up when the available OPcache interned strings buffer size is more than 90% filled. You have a relatively long list of enabled apps, in which case this seems to be not too unexpected. We had one more such report, and another one where Nextcloud was not the only web application running on the same PHP (and hence OPcache) pool.

On Debian with PHP8.0, I suggest to raise it with an own override config module, doubling it to 16 MiB:

# Create an own PHP module with highest priority
cat << '_EOF_' > /etc/php/8.0/mods-available/local-opcache.ini
; Raise OPcache interned strings buffer for Nextcloud
; priority=99
opcache.interned_strings_buffer=16
_EOF_
# Enable the module
phpenmod local-opcache
# Restart Apache2 to apply the new module
systemctl restart apache2

I did a sudo service apache2 restart and it went away but then comes back after a while

This makes sense since the OPcache as well as its interned strings buffer are filled on demand by default, i.e. each script is loaded once requested the first time. When you restart Apache2, the OPcache instance is destroyed and it the buffer is filled while you using the Nextcloud web interface and the individual apps.

Since this seems to be not as rare as I expected, it makes sense that we add this explicitly to the docs: https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache
Also I see a link to the docs from the admin panel is missing. I'll add this as well now that docs and admin panel recommendations match.

Not sure whether it breaks the scope of the docs, but I could also add the steps like above to add a config override, the IMHO preferred way since no package files are touched, so that changes can be reverted easily and package manager upgrades can update the default configuration files without interactive confirmation and loosing customisations. In case, does anyone know whether phpenmod/phpdismod is specific to Debian/Ubuntu/APT or whether these commands are available on other distros, RPM and AUR based ones, and upstream PHP builds as well?

@MichaIng

@MichaIng Sorry I searched everywhere for something regarding this and nothing came up for me I even check the docs and I didn't see anything about a recommended opcache.interned_strings_buffer=16. so I though to submit a bug. I should change my settings from 8 to 16 ?

Okay, no problem, I asked just in case it showed something wrong, e.g. you raised the value already or verified that in fact the buffer is not so full.

I didn't see anything about a recommended opcache.interned_strings_buffer=16.

And this makes sense as it depends on the enabled apps and in case other PHP applications running on the same PHP/OPcache instance. In most cases, the default 8 MiB are sufficient, in rare cases, as far we could see, 16 MiB are required to keep everything stored in cache. But it wouldn't make sense to recommend this to everyone when the additional 8 MiB reserved in total OPcache was just empty in most cases 😉. So the warnings do now show up based on actual current cache usage.

But as I said, it makes sense to mention it in the docs and link that docs section from within the admin panel warnings. I'll raise a PR the next days.

I should change my settings from 8 to 16?

This I would recommend, using e.g. the commands I provided above. Note that this is not mandatory, it just assures that all strings present in Nextcloud's PHP scripts stay cached and hence speeds up access performance a bit. The interned strings buffer is quite a clever implementation which stores repeating strings across all cached scripts just a single time in shared memory and additionally persistent for future requests (which doesn't seem to be the default according to some readings).

All my NC installations were working fine for years, but right after upgrading to 23.0.2 I get these errors on every single installation.

I have the same problem, wenn i raise it to 16 it sayy raise it higher, wenn i raise it to 32 it says raise it higher

Exactly the same. No matter how high I set new limits, error re-appears again and again.

@Copiis

All my NC installations were working fine for years

And it still does. Note that the admin panel warnings/recommendations are nothing mandatory, just a hint what may be improved according to how far Nextcloud is able to check it. But warnings may be wrong or not apply in a particular case, e.g. I keep ignoring warnings about missing imagick (I do not, and do not want to use ImageMagic and Nextcloud works very well with GD), missing bcmath and gmp (I do not use WebAuthn, and those are not required for anything else) and about the missing X-Download-Options header (I do not use Internet Explorer, nor does any other user of my instance). I however understand that one usually wants to have the admin panel free of warnings, which is why I was already requesting to solve the bcmath and gmp recommendations differently, embedding them right into the WebAuthn settings, keep observing the discussion about getting rid of imagick (which is known to have potential security issues with SVG embedded code) and cannot await #20855 to be fully done.

Back to topic: I cannot imagine that one Nextcloud instance is able to use an interned strings buffer of more than 16 MiB, so @Copiis @toxpal are you running only a single Nextcloud instance on that webserver/PHP pool or are there other web applications (which do use PHP)? If it is really just one Nextcloud, can you show your occ app:list, please?

I cannot imagine that one Nextcloud instance is able to use an interned strings buffer of more than 16 MiB

My little Nextcloud has the same issue, too. I used opcache before, but immediately after upgrade from 22 to 23 a couple of days ago I got this opcache warning, too. I tried to get rid of it by doubling interned_strings_buffer from 8 to 16 and now 32, but this warning still exists. It feels like a newly introduced bug. Is there a possibility to roughly estimate the needed buffer space NC needs?

Same as for the others, if Nextcloud is really the only website/application running on that webserver and PHP instance, please post the output of occ app:list so we may find a similarity between your apps which might cause this.

Is there a possibility to roughly estimate the needed buffer space NC needs?

The admin panel does exactly this. It looks how much of the available interned strings buffer is in fact used, and when >90% of available size, it prints the recommendation. If Nextcloud is really the only website, then obviously it requires that much interned strings buffer in your particular case, at least when you want to assure that all strings are kept in cache.

The admin panel does exactly this. It looks how much of the available interned strings buffer is in fact used, and when >90% of available size, it prints the recommendation.

So this check is a new NC 23 feature?

NC v23.0.1 to be precise.

I still could not find any permanent solution for this. Had opcache.interned_strings_buffer set to a huge number like 272 before, warning still comes back after a few days. This warning seems to be misleadingly shown. Restarting PHP-FPM makes the warning disappear for some time, but that's just a workaround. No discovered impact by this warning - except the warning itself, bugging me for months (or almost a year? not even sure when it started).

I still could not find any permanent solution for this. Had opcache.interned_strings_buffer set to a huge number like 272 before, warning still comes back after a few days. This warning seems to be misleadingly shown. Restarting PHP-FPM makes the warning disappear for some time, but that's just a workaround. No discovered impact by this warning - except the warning itself, bugging me for months (or almost a year? not even sure when it started).

I encourage you to find my posted solution above. It continues to work in 10 Nextcloud production environments. In a few of them, I needed to update php version for some apps that required it. The recommendation came back, and I simply entered the configuration again, and went right away. There's sadly clutter on this page which detracts people from being able to solve this. Best, oemb1905.

@Copiis

Tested with the patch up and down, and indeed was running also into the case where the interned strings buffer was exhausted up to the last byte, in which case the recommendation was kept being shown, with the patch not anymore.

@Copiis
Just to rule it out, since I am a little wondering that NC + Wordpress do not use more of the OPcache: Could you:

echo '<?php echo '\''<pre>'\''; print_r(opcache_get_status(false));' > /path/to/webroot/opcache.php

and point your browser to domain.org/opcache.php and paste the output?

@oemb1905

@oemb1905 there's no posted solution above cause there's no other post of you. Linking would help, searching didn't ;-)

I really tried all possible combinations, for the momentI'm just tired and gonna lean back looking forward to see #36278 arriving.

@bcutter

@bcutter
#31223 (comment)

I'd personally use:

opcache.memory_consumption = 64
opcache.interned_strings_buffer = 16

which gives more buffer to the overall OPcache, while the interned strings buffer warnings still won't show up anymore, as already a quarter of overall size (or more).

If you run more than Nextcloud on the same PHP server, in which case you may need to increase opcache.memory_consumption, increase opcache.interned_strings_buffer to keep the same ratio of 4:1 and the recommendation about opcache.interned_strings_buffer hence silent.

($status['memory_usage']['used_memory'] / $status['memory_usage']['free_memory'] > 9)

59835240 / 5189720 = 11.53 > 9 hence the recommendation should now show up.

Yes, the message about the overall OPcache buffer shows up, which is correct as I calculated above.

What does not show up is the message about the interned strings buffer, which has only 16 bytes left free space. It didn't show up in this circumstance since a longer time, when being defined as quarter of overall OPcache or larger (I'm repeating myself, is it really that difficult to understand? 😅). But it did still show up if the interned strings buffer had exactly 0 bytes left free space, which is fixed with the patch.

Btw, your output shows that the interned strings buffer is set to 8 MiB (6 MiB usable) only, in which case, also the interned strings buffer message should show up, since (8 < 64 / 4). But you posted opcache.interned_strings_buffer = 16, so this does not match. Did you change something about the settings in the meantime?

@oemb1905

@oemb1905 there's no posted solution above cause there's no other post of you. Linking would help, searching didn't ;-)

I really tried all possible combinations, for the momentI'm just tired and gonna lean back looking forward to see #36278 arriving.

A simple ctrl-f would do the job, so long as you expanded all historical posts with "Load more" - here ya go as a courtesy: #31223 (comment) ... the settings/values are for my use-case, the important part is the location imho. If it does not help, well best luck, but I've yet to have an instance of mine or others I help continue to complain since. Lmk either way, would love to know.

I'm running nextcloud inside lxc container and had to limit the available memory to resolve this issue (server passed 64 GB memory to the container, now I limit on 8GB). Calculating based on 90% maybe not always best aproach.

How is the system memory allocated to the container related to the size and usage of the PHP instance's OPcache and interned strings buffer? The OPcache size is 128 MiB by default, and I never saw a (single) Nextcloud instance using more than that. And the Nextcloud admin panel warns only when the interned string buffer size is less than 25% than that (and above 90% filled), hence less than 32 MiB in this case. This is all some orders of magnitudes below the system/container memory sizes you are mentioning.

@MichaIng

Hi @MichaIng,
I found this issue by the Error message and

($status['memory_usage']['used_memory'] / $status['memory_usage']['free_memory'] > 9)

gave me the idea it may be worth a try to modify free_memory by reducing container's memory.
Maybe I'm totally missleaded but at least the error message went away 🤷‍♂️

This is data from the PHP OPcache, not the system memory. And reducing RAM size of course reduces free memory, which would increase the chance for the panel recommendation to show up, instead of reducing it 😉. Moreover the particular line you are quoting is not about the interned strings buffer (topic here), but about the overall OPcache memory. This is completely unrelated to host or container system memory. The reason why you do not see the admin panel recommendation currently, is because it takes a while before it reappears, as the interned strings buffer is filled slowly.

Since it is known that in some circumstances for unknown reasons, the interned strings buffer is filled slowly unlimited, we adjusted the setup check to not warn anymore when interned strings buffer is 25% of overall OPcache size already. Hence, if your OPcache size is 128 MiB (default), then set the interned strings buffer size to 32 MiB, and all is good. This has been mentioned allover above. Always good to reach through issues and review the attached pull requests, before drawing false conclusions 😉.

Navigation Menu

Repository navigation

[Bug]: The OPcache interned strings buffer is nearly full #31223

⚠️ This issue respects the following points: ⚠️

Bug description

Steps to reproduce

Expected behavior

Installation method

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

125 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: The OPcache interned strings buffer is nearly full #31223

Description

⚠️ This issue respects the following points: ⚠️

Bug description

Steps to reproduce

Expected behavior

Installation method

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

Activity

szaimen commented on Feb 16, 2022

AndyXheli commented on Feb 16, 2022

MichaIng commented on Feb 16, 2022

AndyXheli commented on Feb 16, 2022

MichaIng commented on Feb 16, 2022

toxpal commented on Feb 17, 2022

toxpal commented on Feb 17, 2022

MichaIng commented on Feb 17, 2022

fantasya-pbem commented on Feb 17, 2022

MichaIng commented on Feb 17, 2022

fantasya-pbem commented on Feb 17, 2022

MichaIng commented on Feb 17, 2022

125 remaining items

bcutter commented on Jan 22, 2023

oemb1905 commented on Jan 22, 2023

MichaIng commented on Jan 22, 2023

bcutter commented on Jan 22, 2023

MichaIng commented on Jan 22, 2023

MichaIng commented on Jan 23, 2023

MichaIng commented on Jan 23, 2023

oemb1905 commented on Jan 24, 2023

hi-ko commented on Feb 5, 2024

MichaIng commented on Feb 5, 2024

hi-ko commented on Feb 5, 2024

MichaIng commented on Feb 5, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions