Friday, November 29, 2013

Some thoughts on large data processing

First be ready for digit/string tricks.

Stata recommended.  SAS sucks.

Why SAS sucks?  Will do a separate post to discuss it.

Need to have a full license of StatTransfer

Be ready to compress data using compress command in Stata.

Use a codebook.

Do not try to append all the data files into one file.  Should put them within one folder and create global code to index them.