This.

05/09/2017 08:22:12 | Rosetta@home | Started upload of rb_09_04_77236_119997__t000__0_C1_SAVE_ALL_OUT_IGNORE_THE_REST_514924_697_0_r181725287_0
05/09/2017 08:22:12 | Rosetta@home | Started upload of cdfdc_SOL_jumping_SAVE_ALL_OUT_514877_4989_0_r1586011663_0
05/09/2017 08:22:14 | Rosetta@home | [error] Error reported by file upload server: [rb_09_04_77236_119997__t000__0_C1_SAVE_ALL_OUT_IGNORE_THE_REST_514924_697_0_r181725287_0] locked by file_upload_handler PID=255
05/09/2017 08:22:14 | Rosetta@home | [error] Error reported by file upload server: [cdfdc_SOL_jumping_SAVE_ALL_OUT_514877_4989_0_r1586011663_0] locked by file_upload_handler PID=255
05/09/2017 08:22:14 | Rosetta@home | Temporarily failed upload of rb_09_04_77236_119997__t000__0_C1_SAVE_ALL_OUT_IGNORE_THE_REST_514924_697_0_r181725287_0: transient upload error
05/09/2017 08:22:14 | Rosetta@home | Backing off 05:09:59 on upload of rb_09_04_77236_119997__t000__0_C1_SAVE_ALL_OUT_IGNORE_THE_REST_514924_697_0_r181725287_0
05/09/2017 08:22:14 | Rosetta@home | Temporarily failed upload of cdfdc_SOL_jumping_SAVE_ALL_OUT_514877_4989_0_r1586011663_0: transient upload error
05/09/2017 08:22:14 | Rosetta@home | Backing off 04:08:07 on upload of cdfdc_SOL_jumping_SAVE_ALL_OUT_514877_4989_0_r1586011663_0
05/09/2017 08:22:15 | Rosetta@home | Started upload of 95dbf_SOL_jumping_SAVE_ALL_OUT_514877_4988_0_r1180513244_0
05/09/2017 08:22:16 | Rosetta@home | [error] Error reported by file upload server: [95dbf_SOL_jumping_SAVE_ALL_OUT_514877_4988_0_r1180513244_0] locked by file_upload_handler PID=-1
05/09/2017 08:22:16 | Rosetta@home | Temporarily failed upload of 95dbf_SOL_jumping_SAVE_ALL_OUT_514877_4988_0_r1180513244_0: transient upload error
05/09/2017 08:22:16 | Rosetta@home | Backing off 00:16:02 on upload of 95dbf_SOL_jumping_SAVE_ALL_OUT_514877_4988_0_r1180513244_0

Same here.

[quote]rb_09_04_77236_119997__t000__0_C1_SAVE_ALL_OUT_IGNORE_THE_REST_514924_697_0_r181725287_0: transient upload error[/quote

+1

---

Same for me.

Tue 05 Sep 2017 17:22:55 SAST | Rosetta@home | Started upload of rb_09_03_77151_119988_ab_stage0_t000___robetta_IGNORE_THE_REST_05_09_514908_71_0_r1550815990_0
Tue 05 Sep 2017 17:22:59 SAST | Rosetta@home | [error] Error reported by file upload server: [rb_09_03_77151_119988_ab_stage0_t000___robetta_IGNORE_THE_REST_05_09_514908_71_0_r1550815990_0] locked by file_upload_handler PID=255
Tue 05 Sep 2017 17:22:59 SAST | Rosetta@home | Temporarily failed upload of rb_09_03_77151_119988_ab_stage0_t000___robetta_IGNORE_THE_REST_05_09_514908_71_0_r1550815990_0: transient upload error
Tue 05 Sep 2017 17:22:59 SAST | Rosetta@home | Backing off 04:48:33 on upload of rb_09_03_77151_119988_ab_stage0_t000___robetta_IGNORE_THE_REST_05_09_514908_71_0_r1550815990_0

Me too
9/5/2017 6:55:09 PM | Rosetta@home | [error] Error reported by file upload server: [6ea159939ab8b54604c3f5fbeadf8c01_C2_docking_big_job_17_08_03_12_20_globalDocking_0_SAVE_ALL_OUT_510695_8_0_r1334300603_0] locked by file_upload_handler PID=255
9/5/2017 6:55:09 PM | Rosetta@home | Temporarily failed upload of 6ea159939ab8b54604c3f5fbeadf8c01_C2_docking_big_job_17_08_03_12_20_globalDocking_0_SAVE_ALL_OUT_510695_8_0_r1334300603_0: transient upload error
9/5/2017 6:55:09 PM | Rosetta@home | Backing off 05:19:22 on upload of 6ea159939ab8b54604c3f5fbeadf8c01_C2_docking_big_job_17_08_03_12_20_globalDocking_0_SAVE_ALL_OUT_510695_8_0_r1334300603_0

I have four of them on two machines, making a total of 8 tasks (AKA units) stuck in the "Uploading" status on the Tasks tab. I didn't see how the second machine got there, but on the first machine there was definitely an interval when only three units were stuck, and the fourth was added later. On the machine I can see now, all four of them are from different projects, with completion times from 4 to 8 hours.

On the Transfers tab, the "Upload: retry in ..." times vary from 2 to 5 hours. Using the Retry Now button individually or collectively fails after about 2 seconds of "active" status.

Not a new problem, but I think this is the first time I've seen it since the major server upgrade a few weeks back.

There was at least one other peculiar behavior, but since the one I can recall right now involves the arbitrary and meaningless deadlines, I file it under "C'est la vie." At least the deadlines continue to appear arbitrary and without meaning from my perspective as a volunteer or donor... Their only significance is the demotivating feeling of well-intended contributions tossed in the bit bucket, which may happen to these frozen-in-Uploading units, too. Sometimes I feel like instead of saying "C'est la vie" I should be saying "Cela signifie la guerre!" (At least in this case I understand the circumstances which caused the deadlines to be missed, so I can basically dismiss the lost hours as a one-time failure affecting 2.5 machines.)

Ah. Just finished another task (AKA work unit) on this machine, and it went to the "Ready to report" status. Going to the Projects tab and clicking Update works as expected. The newly finished task disappears, and the four "Uploading" tasks remain unaffected. Clicking on Retry Now from the Transfers tab fails.

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)

Now up to 5 "Uploading" tasks frozen on this machine with total work time over 30 hours. Doesn't appear to be an immediate threat of lost credit, since the earliest deadline is the 11th, but looking at the other machine... It has also increased to 5 making a total of 10 jammed units. Checked two other machines at hand, and they don't have any yet.

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)

Maybe a restart of server's daemons.....

---

On the Transfers tab, the "Upload: retry in ..." times vary from 2 to 5 hours. Using the Retry Now button individually or collectively fails after about 2 seconds of "active" status.

Not a new problem, but I think this is the first time I've seen it since the major server upgrade a few weeks back.

Same.

If I remember correctly, a solution was found last time on the server side - that shouldn't have solved anything, but it did. If someone searches for "transient" I think that solution will be found.

---

Well, I can report that it has spread to two more computers.

Since I had changed my network configuration recently, I went ahead and tested one machine with an alternate routing. Much slower connection, but no apparent effects on the problem.

There seems to be some inconsistency in how quickly a "Retry Now" from the Transfer tab fails. On this machine, it goes back to the "Upload: retry in ..." status quite quickly, just a few seconds. Other machines remain in "Upload: active" status for a long time.

Quite annoying, but not surprising or anything... Which leads to the long history of project struggles. I do recall something about a network security configuration problem at the Baker Lab sign. If this is similar, then it took them about a week to communicate the nature of the problem to the university's network people and get the fix, whatever it was.

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)

All seems to be working for me this morning.
Units finished and retry queued on Sep 5th were credited on the 6th.
One way arrow of time strikes again ... oh well.
Anybody know that happened/resolution?

All seems to be working for me this morning.
Units finished and retry queued on Sep 5th were credited on the 6th.
One way arrow of time strikes again ... oh well.
Anybody know that happened/resolution?

Mine cleared up at almost exactly the same time - 2 minutes prior to this post

---

"Which leads to the long history of project struggles."

We've been up since late June with the new server front and back ends without any significant issues (knock on wood). This has been the first significant issue since and may or may not have been related to network and power instability here at the UW recently. The file locking logic in the upload handler started to fail for the majority of upload requests. We rebooted the web servers and filesystem but that didn't fix the issue. We had to modify the source code to comment out the file locking logic and rebuild the upload handler. This appears to have fixed the issue. The file locking logic is not necessary for our system and things appear to be back to normal.

On a positive note, I think our project has a long history of success including research from our lab being runner-up to Science magazine's breakthrough of 2016 for protein design, success in using co-evolution sequence data from meta-genomes to determine new protein structures at a cost significantly less than structural genomics initiatives, and designing/modeling small cyclic peptides with non-canonical amino acids (much of which was modeled on mobile android devices), and more.

Thanks for the update. :-)

The quote sounds like one of my asides... Anyway, all seems back to normal and mostly I don't care much these days. I continue to bubble with imaginary constructive suggestions and continue to feel the world at large will do what it darn well pleases. Just saw another one implemented today about 30 years after I first wrote about it... (If my memory wasn't so darned selective I would be better at remembering all my erroneous ideas, too.)

---

#1 Freedom = (Meaningful - Constrained) Choice{5} != (Beer^3 | Speech)