json-machine

Very easy to use and memory efficient drop-in replacement for inefficient iteration of big JSON files or streams for PHP >=7.2. See TL;DR. No dependencies in production except optional ext-json. README in sync with the code

NEW in version 1.2.0 - Recursive iteration

• TL;DR
• Introduction
• Parsing JSON documents
  • Parsing a document
  • Parsing a subtree
  • Parsing nested values in arrays
  • Parsing a single scalar value
  • Parsing multiple subtrees
  • Recursive iteration
  • What is JSON Pointer anyway?
• Options
• Parsing streaming responses from a JSON API
  • GuzzleHttp
  • Symfony HttpClient
• Tracking the progress
• Decoders
  • Available decoders
• Error handling
  • Catching malformed items
• Parser efficiency
  • Streams / files
  • In-memory JSON strings
• Troubleshooting
  • "I'm still getting Allowed memory size ... exhausted"
  • "That didn't help"
  • "I am still out of luck"
• Installation
• Development
  • Non containerized
  • Containerized
• Support
• License

TL;DR

 $user) {
    // just process $user as usual
    var_dump($user->name);
}

Random access like $users[42] is not yet possible. Use above-mentioned foreach and find the item or use JSON Pointer.

Counting the items is possible via iterator_count($users). Remember it will still have to internally iterate the whole thing to get the count and thus will take about the same time as iterating it and counting in a loop.

Requires ext-json if used out of the box but doesn't if a custom decoder is used. See Decoders.

Follow CHANGELOG.

Introduction

JSON Machine is an efficient, easy-to-use and fast JSON stream/pull/incremental/lazy (whatever you name it) parser based on generators developed for unpredictably long JSON streams or documents. Main features are:

• Constant memory footprint for unpredictably large JSON documents.
• Ease of use. Just iterate JSON of any size with foreach. No events and callbacks.
• Efficient iteration on any subtree of the document, specified by JSON Pointer
• Speed. Performance critical code is optimized for speed with JIT in mind, and uses native json_decode to decode JSON document items by default. See Decoders.
• Parses not only streams but any iterable that produces JSON chunks.
• Thoroughly tested. More than 200 tests and 1000 assertions.

Parsing JSON documents

Parsing a document

Let's say that fruits.json contains this huge JSON document:

// fruits.json
{
    "apple": {
        "color": "red"
    },
    "pear": {
        "color": "yellow"
    }
}

It can be parsed this way:

 $data) {
    // 1st iteration: $name === "apple" and $data->color === "red"
    // 2nd iteration: $name === "pear" and $data->color === "yellow"
}

Parsing a json array instead of a json object follows the same logic. The key in a foreach will be a numeric index of an item.

If you prefer JSON Machine to return arrays instead of objects, use new ExtJsonDecoder(true) as a decoder.

 new ExtJsonDecoder(true)]);

Parsing a subtree

If you want to iterate only results subtree in this fruits.json:

// fruits.json
{
    "results": {
        "apple": {
            "color": "red"
        },
        "pear": {
            "color": "yellow"
        }
    }
}

use JSON Pointer /results as pointer option:

 '/results']);
foreach ($fruits as $name => $data) {
    // The same as above, which means:
    // 1st iteration: $name === "apple" and $data->color === "red"
    // 2nd iteration: $name === "pear" and $data->color === "yellow"
}

Note:

Value of results is not loaded into memory at once, but only one item in results at a time. It is always one item in memory at a time at the level/subtree you are currently iterating. Thus, the memory consumption is constant.

Parsing nested values in arrays

The JSON Pointer spec also allows to use a hyphen (-) instead of a specific array index. JSON Machine interprets it as a wildcard which matches any array index (not any object key). This enables you to iterate nested values in arrays without loading the whole item.

Example:

// fruitsArray.json
{
    "results": [
        {
            "name": "apple",
            "color": "red"
        },
        {
            "name": "pear",
            "color": "yellow"
        }
    ]
}

To iterate over all colors of the fruits, use the JSON Pointer "/results/-/color".

 '/results/-/color']);

foreach ($fruits as $key => $value) {
    // 1st iteration:
    $key == 'color';
    $value == 'red';
    $fruits->getMatchedJsonPointer() == '/results/-/color';
    $fruits->getCurrentJsonPointer() == '/results/0/color';

    // 2nd iteration:
    $key == 'color';
    $value == 'yellow';
    $fruits->getMatchedJsonPointer() == '/results/-/color';
    $fruits->getCurrentJsonPointer() == '/results/1/color';
}

Parsing a single scalar value

You can parse a single scalar value anywhere in the document the same way as a collection. Consider this example:

// fruits.json
{
    "lastModified": "2012-12-12",
    "apple": {
        "color": "red"
    },
    "pear": {
        "color": "yellow"
    },
    // ... gigabytes follow ...
}

Get the scalar value of lastModified key like this:

 '/lastModified']);
foreach ($fruits as $key => $value) {
    // 1st and final iteration:
    // $key === 'lastModified'
    // $value === '2012-12-12'
}

When parser finds the value and yields it to you, it stops parsing. So when a single scalar value is in the beginning of a gigabytes-sized file or stream, it just gets the value from the beginning in no time and with almost no memory consumed.

The obvious shortcut is:

 '/lastModified']);
$lastModified = iterator_to_array($fruits)['lastModified'];

Single scalar value access supports array indices in JSON Pointer as well.

Parsing multiple subtrees

It is also possible to parse multiple subtrees using multiple JSON Pointers. Consider this example:

// fruits.json
{
    "lastModified": "2012-12-12",
    "berries": [
        {
          "name": "strawberry", // not a berry, but whatever ...
          "color": "red"
        },
        {
          "name": "raspberry", // the same ...
          "color": "red"
        }
    ],
    "citruses": [
      {
          "name": "orange",
          "color": "orange"
      },
      {
          "name": "lime",
          "color": "green"
      }
    ]
}

To iterate over all berries and citrus fruits, use the JSON pointers ["/berries", "/citrus"]. The order of pointers does not matter. The items will be iterated in the order of appearance in the document.

…

Recursive iteration

Use RecursiveItems instead of Items when the JSON structure is difficult or even impossible to handle with Items and JSON pointers or the individual items you iterate are too big to handle. On the other hand it's notably slower than Items, so bear that in mind.

When RecursiveItems encounters a list or dict in the JSON, it returns a new instance of itself which can then be iterated over and the cycle repeats. Thus, it never returns a PHP array or object, but only either scalar values or RecursiveItems. No JSON dict nor list will ever be fully loaded into memory at once.

Let's see an example with many, many users with many, many friends:

// users.json
[
  {
    "username": "user",
    "e-mail": "[email protected]",
    "friends": [
      {
        "username": "friend1",
        "e-mail": "[email protected]"
      },
      {
        "username": "friend2",
        "e-mail": "[email protected]"
      }
    ]
  }
]

 $value) {
        if ($field === 'friends') {
            /** @var $value RecursiveItems */
            foreach ($value as $friend) {
                /** @var $friend RecursiveItems */
                foreach ($friend as $friendField => $friendValue) {
                    $friendField == 'username';
                    $friendValue == 'friend1';
                }
            }
        }
    }
}

If you break an iteration of such lazy deeper-level (i.e. you skip some "friends" via break) and advance to a next value (i.e. next user), you will not be able to iterate it later. JSON Machine must iterate it in the background to be able to read next value. Such an attempt will result in closed generator exception.

Convenience methods of RecursiveItems

• toArray(): array If you are sure that a certain instance of RecursiveItems is pointing to a memory-manageable data structure (for example, $friend), you can call $friend->toArray(), and the item will materialize into a plain PHP array.

• advanceToKey(int|string $key): scalar|RecursiveItems When searching for a specific key in a collection (for example, 'friends' in $user), you do not need to use a loop and a condition to search for it. Instead, you can simply call $user->advanceToKey("friends"). It will iterate for you and return the value at this key. Calls can be chained. It also supports array like syntax for advancing to and getting following indices. So $user['friends'] would be an alias for $user->advanceToKey('friends'). Calls can be chained. Keep in mind that it's just an alias - you won't be able to random-access previous indices after using this directly on RecursiveItems. It's just a syntax sugar. Use toArray() if you need random access to indices on a record/item.

The previous example could thus be simplified as follows:

advanceToKey('friends')
        /** @var $friend RecursiveItems */
        $friendArray = $friend->toArray();
        $friendArray['username'] === 'friend1';
    }
}

Chaining allows you to do something like this:

 true]);
foreach ($fruits as $name => $data) {
    echo 'Progress: ' . intval($fruits->getPosition() / $fileSize * 100) . ' %'; 
}

Decoders

Items::from* functions also accept decoder option. It must be an instance of JsonMachine\JsonDecoder\ItemDecoder. If none is specified, ExtJsonDecoder is used by default. It requires ext-json PHP extension to be present, because it uses json_decode. When json_decode doesn't do what you want, implement JsonMachine\JsonDecoder\ItemDecoder and make your own.

Available decoders

• ExtJsonDecoder - Default. Uses json_decode to decode keys and values. Constructor has the same parameters as json_decode.

• PassThruDecoder - Does no decoding. Both keys and values are produced as pure JSON strings. Useful when you want to parse a JSON item with something else directly in the foreach and don't want to implement JsonMachine\JsonDecoder\ItemDecoder. Since 1.0.0 does not use json_decode.

Example:

 new PassThruDecoder]);

• ErrorWrappingDecoder - A decorator which wraps decoding errors inside DecodingError object thus enabling you to skip malformed items instead of dying on SyntaxError exception. Example:

 new ErrorWrappingDecoder(new ExtJsonDecoder())]);
foreach ($items as $key => $item) {
    if ($key instanceof DecodingError || $item instanceof DecodingError) {
        // handle error of this m