Review data storage/versioning

[Peter Powers]

Determine what data should and should not be stored in repository.

(e.g.) org.opensha.sha.simulators.eqsim_v04.NCAL_Ward.out.txt is 110MB and wreaks havoc on merges.

[Kevin Milner]

I did some searching to find a standardised way of downloading and caching data files via http requests, but didn't find any good matches. If we want to download our data as requested (like simulator files, and CEUS input files), then we might need a home grown solution. Peter, do you know of any alternatives?

What I'm thinking of would be similar to a web browser's cache. When a resource is requested, it first checks if it's already cached, and then either downloads it if needed, or uses the cached version. We would have to get a little fancy though, with some way to check if the remote file has changed. Possibly comparing md5sums?

[Peter Powers]

Replying to kmilner:

I did some searching to find a standardised way of downloading and caching data files via http requests, but didn't find any good matches. If we want to download our data as requested (like simulator files, and CEUS input files), then we might need a home grown solution. Peter, do you know of any alternatives?

What I'm thinking of would be similar to a web browser's cache. When a resource is requested, it first checks if it's already cached, and then either downloads it if needed, or uses the cached version. We would have to get a little fancy though, with some way to check if the remote file has changed. Possibly comparing md5sums?

Talking with Ned, we should leave this alone for now and wait and see what the reqs for UCERF3 might be.