Request: Make tasks share the database

Message boards : Number crunching : Request: Make tasks share the database

To post messages, you must log in.

AuthorMessage
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 90370 - Posted: 16 Feb 2019, 22:53:39 UTC - in response to Message 90369.  
Last modified: 16 Feb 2019, 22:55:21 UTC

When running 8 work units at a time, with the 8 hour length (so three downloads per core per day), then downloading the database adds 400 x 8 x 3 = 9.6 GB/day. And in operation (not including downloads), I am seeing writes of 11 GB/day for six Rosettas (four of 3.78 and two of 4.07) on my i7-4771 under Win 7 64-bit. So the total writes would be about 20 GB/day on my machine.

Doubling that for running 16 cores is still a relatively small amount, and not a problem for SSD life. And you can run longer work units if you want to reduce it further.
ID: 90370 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 90371 - Posted: 17 Feb 2019, 2:41:21 UTC - in response to Message 90370.  

And I just checked it on an Ubuntu 16.04 machine, where I am running Rosetta 3.78 on six cores of an i7-3770. The writes (averaged over one hour) are 30 kB_WRTN/s according to iostat. That works out to only 2.6 GB/day. Add to that the downloads, and it comes out to even less than for Windows.
ID: 90371 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Link
Avatar

Send message
Joined: 4 May 07
Posts: 356
Credit: 382,349
RAC: 0
Message 90374 - Posted: 17 Feb 2019, 10:12:33 UTC - in response to Message 90369.  

My result is that the above behavior can easily cause terabytes to be written where some hundred megabytes should be enough.

Besides the fact, that current SSDs can take petabytes (https://techreport.com/review/27436/the-ssd-endurance-experiment-two-freaking-petabytes), so terabytes are not really something you need to think much about, you can set the target CPU run time in your Rosetta@home preferences to 24 hours. That will cut down the writes to 1/3 of the standard value.

But yes, in general I agree with you, two persistent databases would be nice to have, that would speed up the start up of each WU, specially on systems with HDD.
.
ID: 90374 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Link
Avatar

Send message
Joined: 4 May 07
Posts: 356
Credit: 382,349
RAC: 0
Message 90376 - Posted: 17 Feb 2019, 14:49:42 UTC - in response to Message 90375.  

Larger SSDs are better with respect to that but cost a lot more and in the end 98% of space is unused.

This space isn't really unused, it's used for wear leveling. More free space on SSD means longer life time, because the data won't be written always to the same few empty cells.


To achieve that I decided to follow the manufacturer's recommendations, which for the devices in question mostly are between 40 and 70 TB of total writes, (...) 10 year life time, (...) 9.6 GB a day (...) That extrapolates to 3.5 TB/year

So that means between 10 and 20 years life time (when we think about the writes), much longer than any HDD will last. And as I said, you can cut that down to 3.2GB/day by changing your rosetta@home preferences which will give you up to 60 years life time. The SSD will very likely fail long time before that for some other random reason and not because of the data written to it by the rosetta application, specially if you plan to buy a cheap one.
.
ID: 90376 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Request: Make tasks share the database



©2025 University of Washington
https://www.bakerlab.org