Systems Programming

Processing Gigabytes of JSON: Why Streaming is the Only Way to Scale The Massive Config Audit In large-scale cloud or storage environments, the “state of the world” is often exported as a giant JSON dump (e.g., an inventory of every virtual disk, volume, and node in a cluster). These files can be tens of gigabytes when uncompressed. If we try to open these with a standard JSON library, our application will crash because those libraries typically build a DOM (Document Object Model)—a tree structure in RAM that is often 5-10x larger than the file itself. ...

Processing Billions of Integers: A Lesson in Memory Efficiency and Bit Manipulation The Challenge In storage systems, we often deal with raw binary dumps of metadata tables. When debugging consistency issues, you may need to scan billions of 32-bit identifiers to find duplicates or unique entries. Because these files can be massive, loading them into a standard hash map is impossible. The task seems simple on the surface: given a binary file containing 1 billion 32-bit integers, count how many are unique and how many appear exactly once. ...