Its a constant challenge to keep many of the systems that I help develop up to date with Government data sources. One of the most awkward is the LARS (Learning Aims Reference Service) database. (https://hub.fasst.org.uk/Learning%20Aims/Downloads/Pages/default.aspx)
It’s updated several times a month, and is only available for download in .mdb, .csv and .psv files – none of which are even remotely useful for integrating with existing databases in a practical way.
We have developed a piece of software that converts the MDB file into a MySQL compatible .sql file so that it can be easily run to import the full database. Everything we do is in the cloud, and so we needed a .sql file that we could just run against an Amazon RDS instance.
Because we needed to know what database to put the data in, we make a separate database called “lars” on our existing database server, and then run a script to simply copy all data across, thus meaning that there is zero interruption to live system operations whilst the database update happens.