system_list() memory usage

Updated: Comment #88

Problem/Motivation

With system_list(), memory usage increases significantly when large numbers of modules are enabled.

Issue #887870: Make system_list() return full module records triggered this memory spike, since full module records are returned, rather than just module names. Since #328357: Allow module .info files to add CSS/JS allowed module .info files to pull in CSS/JS, this can lead to very high memory usage.

Examples:

• 31 modules enabled: Roughly 400kb of memory used
• 90 modules enabled: Roughly 676kb of memory used

Proposed resolution

A lot of information in the system table isn't needed with system_list(). Instead, we can just store the array directly with the relevant info and improve how the cache is handled.

Here is an example of the performance improvements:

Before Patch

• Cache clear: 38206ms, 188.5MB RAM
• Cached page load: 315ms, 15.25MB RAM

After Patch

• Cache clear: 35875ms, 186.5MB RAM
• Cached page load: 308ms, 14.5MB RAM

Please note that this patch requires update.php to be run after it is applied.

Remaining tasks

Currently this patch is queued for re-testing.

User interface changes

None

API changes

Rather than loading full module records into the system list, only the requested module information is returned.

Related Issues

• #887870: Make system_list() return full module records
• #328357: Allow module .info files to add CSS/JS
• #624848: Allow to retrieve a list of modules defining a certain .info file property
• #402896: Introduce DrupalCacheArray and use it for drupal_get_schema()
• #1503224: Cleanup module_list()
• #1493098: Convert cron settings to configuration system
• #1029460: Find a better hook than hook_init() for drupal_add_js() and drupal_add_css()
• #1861676: Remove stylesheets[] and scripts[] .info file property support for modules

Original report by [catch]

system_list() with 31 modules enabled uses around 400kb of memory. On a D7 site with 90 modules installed this increases to 676kb - 13% of total memory usage on that site.

This is due to #887870: Make system_list() return full module records - we did benchmarks there but no-one including me actually checked the memory implications.
We have to keep info there, which is the biggest issue, due to css/js additions in system_init(), I didn't like that change but it's too late to change now at least for D7. Although with a small API change we could strip some of the other info keys, but that needs to happen in another issue. edit: that's out of the question for D7 too, we're using things like configuration_path via system_get_info() too.

However there's a lot of crap in system table that we don't need, so let's try to cut that down.

Here's how block module looks in HEAD:

            [block] => stdClass Object
                (
                    [filename] => modules/block/block.module
                    [name] => block
                    [type] => module
                    [owner] =>
                    [status] => 1
                    [bootstrap] => 0
                    [schema_version] => 7007
                    [weight] => -5
                    [info] => Array
                        (
                            [name] => Block
                            [description] => Controls the visual building blocks a page is constructed with. Blocks are boxes of content rendered into an area, or region, of a web page.
                            [package] => Core
                            [version] => 7.0-dev
                            [core] => 7.x
                            [files] => Array
                                (
                                    [0] => block.test
                                )

                            [configure] => admin/structure/block
                            [dependencies] => Array
                                (
                                )

                            [php] => 5.2.4
                            [bootstrap] => 0
                        )

                )

Instead of this we can just store the array directly.

These numbers include the entire memory usage from system_info() - including bootstrap, themes and filenames, but the saving is only against modules, still very measurable though:

With 30 modules:
before patch: 415,016 / peak: 495,464

After patch: 330,240 / peak: 394,712

With 90 modules:

Before patch: 691,620 / peak: 829,164

After patch: 569,200 / 684,872

So that's > 100kb off inclusive peak memory usage, and between 85kb to 120kb off inclusive memory usage. With 150 or 200 contrib modules installed we could be looking at more like 200kb of savings.

This is going to be a bit tricky with upgrades due to the array being used in full bootstrap, you can get more modest memory savings from keeping it as an object with just the info property, that's about 15kb more expensive with 30 modules but less of a change. Uploading both patches for now.