Skip to content
Search

Last updated

Read a large JSON file in PHP without running out of memory

Let's say a partner sends us their orders as one JSON file: a million orders, 110Mb. We need the revenue per country, and our PHP process runs with the default memory_limit of 128Mb.

The file

First we need a file to work with, so let's generate one. Set $rows to 1_000_000 to get the 110Mb version.

<?php

declare(strict_types=1);

$rows = 1_000;
$file = fopen(__DIR__ . '/orders.json', 'w');

fwrite($file, "[\n");

for ($id = 1; $id <= $rows; $id++) {
    fwrite($file, json_encode([
        'id' => $id,
        'email' => "customer{$id}@example.com",
        'country' => ['PL', 'US', 'DE', 'FR', 'GB'][$id % 5],
        'amount' => ($id % 1000) / 10,
        'created_at' => gmdate('Y-m-d H:i:s', 1_700_000_000 + $id * 60),
    ]) . ($id < $rows ? ",\n" : "\n"));
}

fwrite($file, "]\n");
fclose($file);

echo "orders.json: {$rows} orders\n";
orders.json: 1000 orders

Read a JSON file with json_decode

There is one way to read JSON in PHP that everybody knows: file_get_contents and json_decode.

<?php

declare(strict_types=1);

$orders = json_decode(file_get_contents(__DIR__ . '/orders.json'), true, flags: JSON_THROW_ON_ERROR);
$revenue = [];

foreach ($orders as $order) {
    $revenue[$order['country']] = ($revenue[$order['country']] ?? 0.0) + $order['amount'];
}

arsort($revenue);

foreach ($revenue as $country => $total) {
    printf("%s %.2f\n", $country, $total);
}
GB 10030.00
FR 10010.00
DE 9990.00
US 9970.00
PL 9950.00

With a thousand orders it works. With a million orders it ends like this:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes)

Why json_decode ends with "Allowed memory size exhausted"

json_decode can't read a part of a file. It needs the whole document as one string, and then it builds a PHP array for every order in it. For our file that is 778Mb, seven times the size of the file.

With a CSV file we could switch to fgetcsv and read it row by row. There is nothing like that for JSON. PHP has no function that reads a JSON file one element at a time, so we would have to write our own parser.

Read a large JSON file with Flow

Flow needs PHP 8.3 or newer.

composer require flow-php/etl flow-php/etl-adapter-json
<?php

declare(strict_types=1);

use function Flow\ETL\Adapter\JSON\from_json;
use function Flow\ETL\DSL\{data_frame, ref, sum, to_output};

require __DIR__ . '/vendor/autoload.php';

data_frame()
    ->read(from_json(__DIR__ . '/orders.json'))
    ->groupBy([ref('country')])
    ->aggregate(sum(ref('amount')))
    ->sortBy(ref('amount_sum')->desc())
    ->write(to_output(truncate: false))
    ->run();
+---------+--------------+
| country |   amount_sum |
+---------+--------------+
|      GB | 10030.000000 |
|      FR | 10010.000000 |
|      DE |  9990.000000 |
|      US |  9970.000000 |
|      PL |  9950.000000 |
+---------+--------------+
5 rows

from_json reads the file one order at a time, so it doesn't matter how big the file is. groupBy with sum adds up the amounts per country, and sortBy puts the biggest one first.

How much memory does a large JSON file take in PHP

PHP 8.3, memory_limit=128M:

Orders File size Code Peak memory Time
1,000,000 110Mb json_decode fatal error, needs 778Mb -
1,000,000 110Mb Flow 22Mb 2.6s
1,000,000 110Mb Flow with the flow_php extension 20Mb 1.0s
3,000,000 335Mb json_decode fatal error, needs 2353Mb -
3,000,000 335Mb Flow 22Mb 6.8s
3,000,000 335Mb Flow with the flow_php extension 20Mb 2.2s

Three times more orders cost Flow nothing. They cost json_decode 1575Mb.

flow_php is an optional PHP extension. The script stays the same, without a single change.

To check it on your machine, add this line at the end of any of the scripts above:

printf("peak memory: %.1f MB\n", memory_get_peak_usage(true) / 1024 / 1024);

Found a typo or an outdated section? Edit this guide on GitHub

scroll back to top