Datacomp: Locally Independent Adaptive Compression for Real-World Systems

Non-lossy compression can save time and energy during communication if the cost to compress and send input is less than the cost of sending it uncompressed. Unfortunately, compression can also degrade performance, no single method is always beneficial, and outcomes depend on many factors. As a result, compression choices in real systems are coarsely grained and manually controlled, resulting in suboptimal or even poor performance. Adaptive Compression (AC) systems make compression choices dynamically to optimize utility. Existing AC systems are limited in ways that reduce their suitability for general-purpose computers. Datacomp is an AC system that operates locally and includes no significant hard-coded knowledge. Using real-world data, a broad range of environments and the Comptool "AC Oracle," we show that Datacomp's performance is equivalent or close to the ideal at bandwidths between 1-100Mbit/s, even when static strategies are suboptimal or more costly than no compression. While Datacomp struggles to perform well at 1Gbit/s, understanding why illustrates important challenges for AC systems and suggests solutions.