Opened 13 years ago
Last modified 13 years ago
#127 new task
Review data storage/versioning
| Reported by: | Peter Powers | Owned by: | Peter Powers |
|---|---|---|---|
| Priority: | minor | Milestone: | OpenSHA 2.0 |
| Component: | sha | Version: | |
| Keywords: | Cc: |
Description
Determine what data should and should not be stored in repository.
(e.g.) org.opensha.sha.simulators.eqsim_v04.NCAL_Ward.out.txt is 110MB and wreaks havoc on merges.
Change History (3)
comment:1 follow-up: 2 Changed 13 years ago by
comment:2 Changed 13 years ago by
Replying to kmilner:
I did some searching to find a standardised way of downloading and caching data files via http requests, but didn't find any good matches. If we want to download our data as requested (like simulator files, and CEUS input files), then we might need a home grown solution. Peter, do you know of any alternatives?
What I'm thinking of would be similar to a web browser's cache. When a resource is requested, it first checks if it's already cached, and then either downloads it if needed, or uses the cached version. We would have to get a little fancy though, with some way to check if the remote file has changed. Possibly comparing md5sums?
Talking with Ned, we should leave this alone for now and wait and see what the reqs for UCERF3 might be.
comment:3 Changed 13 years ago by
| Milestone: | OpenSHA 1.2 → OpenSHA 2.0 |
|---|
I did some searching to find a standardised way of downloading and caching data files via http requests, but didn't find any good matches. If we want to download our data as requested (like simulator files, and CEUS input files), then we might need a home grown solution. Peter, do you know of any alternatives?
What I'm thinking of would be similar to a web browser's cache. When a resource is requested, it first checks if it's already cached, and then either downloads it if needed, or uses the cached version. We would have to get a little fancy though, with some way to check if the remote file has changed. Possibly comparing md5sums?