Processing a stream of chunks of JSON with ijson

A follow-up to Using OpenAI functions and their Python library for data extraction and Using the ChatGPT streaming API from Python. If I have a stream of chunks of a larger JSON document, how can I output full individual JSON objects as soon as they are available?

My completed JSON will look like this:

  "items": [
      "title": "Classic and Custom Car Show",
      "date": "Sat, Sep 2",
      "venue_name": "1700 W Hillsdale Blvd • San Mateo, CA",
      "start_time": "9:00 AM"
      "title": "【衛蘭、王君馨、衛詩、鍾舒漫、吉中鳴牧師】MMO AINEO LIVE 愛你喲!佈道會(三藩市站)",
      "date": "Sat, Aug 26",
      "venue_name": "San Mateo Performing Arts Center • San Mateo, CA",
      "start_time": "7:30 PM"
      "title": "San Francisco Small Business Expo 2023",
      "date": "Fri, Aug 18",
      "venue_name": "Hyatt Regency San Francisco Airport • Burlingame, CA",
      "start_time": "10:00 AM"
      "title": "Stanford Genetics Conference on Structural Variants and DNA Repeats",
      "date": "Thu, Sep 7",
      "venue_name": "Stanford Center for Academic Medicine • Palo Alto, CA",
      "start_time": "8:00 AM"
      "title": "DJ QUIK / ROJAI / Z-MAN / DJ SAURUS",
      "date": "Fri, Aug 18",
      "venue_name": "The Longboard Margarita Bar • Pacifica, CALIFORNIA",
      "start_time": "9:00 PM"
      "date": "Sat, Aug 26",
      "venue_name": "Jerry Garcia Amphitheater • San Francisco, CA",
      "start_time": "2:30 PM"

If that's going to arrive as a sequence of chunks, how can I display those items as soon as they become available?

After much experimentation I figured out this recipe using ijson:

import ijson
import json
import time

chunks = [
    '{\n  "items": [\n    {\n      "ti',
    'tle": "Classic and Custom Car ',
    'Show",\n      "date": "Sat, Sep',
    ' 2",\n      "venue_name": "1700',
    " W Hillsdale Blvd • San Mateo,",
    ' CA",\n      "start_time": "9:0',
    '0 AM"\n    },\n    {\n      "titl',
    'e": "【衛蘭、王君馨、衛詩、鍾舒漫、吉中鳴牧師】MMO ',
    'AINEO LIVE 愛你喲!佈道會(三藩市站)",\n   ',
    '   "date": "Sat, Aug 26",\n    ',
    '  "venue_name": "San Mateo Per',
    "forming Arts Center • San Mate",
    'o, CA",\n      "start_time": "7',
    ':30 PM"\n    },\n    {\n      "ti',
    'tle": "San Francisco Small Bus',
    'iness Expo 2023",\n      "date"',
    ': "Fri, Aug 18",\n      "venue_',
    'name": "Hyatt Regency San Fran',
    "cisco Airport • Burlingame, CA",
    '",\n      "start_time": "10:00 ',
    'AM"\n    },\n    {\n      "title"',
    ': "Stanford Genetics Conferenc',
    "e on Structural Variants and D",
    'NA Repeats",\n      "date": "Th',
    'u, Sep 7",\n      "venue_name":',
    ' "Stanford Center for Academic',
    ' Medicine • Palo Alto, CA",\n  ',
    '    "start_time": "8:00 AM"\n  ',
    '  },\n    {\n      "title": "DJ ',
    'US",\n      "date": "Fri, Aug 1',
    '8",\n      "venue_name": "The L',
    "ongboard Margarita Bar • Pacif",
    'ica, CALIFORNIA",\n      "start',
    '_time": "9:00 PM"\n    },\n    {',
    '\n      "title": "DUE SOUTH W/ ',
    'ING ISIS (FREE!)",\n      "date',
    '": "Sat, Aug 26",\n      "venue',
    '_name": "Jerry Garcia Amphithe',
    'ater • San Francisco, CA",\n   ',
    '   "start_time": "2:30 PM"\n   ',
    " }\n  ]\n}",

events = ijson.sendable_list()
coro = ijson.items_coro(events, "items.item")

seen_events = set()

for chunk in chunks:
    if events:
        # Any we have not seen yet?
        unseen_events = [e for e in events if json.dumps(e) not in seen_events]
        if unseen_events:
            for event in unseen_events:

You create an ijson coroutine, then send it chunks of JSON data. It will write new items to the sendable_list() as soon as they are available.

The hardest part to figure out was this:

coro = ijson.items_coro(events, "items.item")

This is the syntax to indicate that I'm interested in the array items in the {"items": [...]} object.

The confusing part is that item here is a reserved word in ijson which means "individual items of an array". I'm not sure what you are meant to do if your nested object also has a key called "item" - I guess work to avoid that situation from coming up!

This works though. The above code, when executed, prints out each of the nested objects one at a time, with a 1 second sleep between each one.

Created 2023-08-15T18:07:38-07:00 · Edit