Using SQLite to Bypass the 2 GB .DBF Filesize Limit

The .DBF file format has been a standard for desktop applications for over 30 years. Pretty much every data-oriented application can read and write to a .DBF file. Back in the day when the format was created, the days of 180 kb floppies, sometime later surpassed by 1.44 MB 3 1/2 inch disks, and, eventually those monster hard disks that held hundreds of or even a thousand megabytes, its 2 GB limit was not a concern.

Even in the mid-2010s, the xBase family of applications are excellent tools for manipulating data. One common use is to import data from other formats, such as text files, and, once stored in a .DBF, slice and dice that data as desired. Over the past decade, though, it's become commonplace to run into data sets exceeding the 2 GB limit. This can happen due to accumulation of data over decades of use of an application or dumps from large SQL databases. Once such a data dump grows larger than two gigabytes, though, dealing with them with the Xbase tools becomes a problem, since they can't be directly imported into a DBF.

What is needed is intermediate data storage that can import a raw text file of any size and then allow the user to slice and dice the data as needed. Enter SQLite. This book will show you how to use SQLite, a fast, free, high-quality database for such intermediate data storage.