BUG - Ultra map processing times out before uploading anything when making a large set of maps (card 3404)

[pleasant-menlo]

Description of the issue

On 2025-12-11 commit 99930fa split the main Ultra L3 job into four separate jobs ("u45-maps", "u90-maps", "ulc-sp-maps", and "ulc-nsp-maps") in order to keep the individual job runtime lower than the timeout. However, the latest runs of "u45-maps", "u90-maps", and "ulc-sp-maps" have all timed out after 3 hours. Because the processor only uploads its outputs after all maps are completed, nothing is uploaded and we restart from the beginning on the next day.

Steps to reproduce the issue

See the jobs which fail after 3.01 hours at https://imap-mission.com/team/processing-jobs#%7B%22instrumentEtc%22:%5B%22ultra%22,%22l3%22%5D%7D

Expected vs Actual behavior

Each output should be uploaded as soon as it is completed. Even if the overall job times out, this will allow us to save our progress and ensure that all maps will eventually be completed (as long as the reprocessing rate is not too high).

Additional notes, affected areas, and suggested fixes

Some other options that could be considered for improving performance and allowing these jobs to complete:

1. Increase the timeout
2. Split into more jobs
3. Use shared HEALPix intermediate maps to produce each set of 2, 4, and 6 degree rectangular maps. (Currently each pixel size independently generates the same HEALpix L2 and SP-corrected maps)
4. Improve the performance of the code which generates rectangular maps from HEALPix.

[lacoak21]

@pleasant-menlo we have purposely blocked files from being uploaded before the job has finished processing because this complicates the version determination and job deduping logic. I think the solution would be to increase the timeout of these jobs. @bryan-harter what do you think?

[jtniehof]

3 or 4 sound worth investigating. Can you do a quick timebox and profile on those two options to figure 1. what kind of benefit we might get; 2. how much effort it might take?

[pleasant-menlo]

Some rough numbers on a local machine:

A set of 2/4/6 degree versions of a map takes 480s of l2/sp processing (320s of which is repeated between pixel sizes) plus 294s healpix-> rectangular conversion = 774s total.

Option 3 would save us 320s. We have some plans about saving the results of each stage in a temporary directory indexed by a hash of the inputs. When running another pixel size of the same map, the cached result can be used.

We looked into how we could speed up the conversion to rectangular, and found that there was debug logging that was taking a noticeable amount of time. Only calculating strings when they will be logged shaves off about 50s when creating a set of 2/4/6 degree maps. We will create a pull request to address this change. Other than that, we didn't find much that can be easily optimized without changing the algorithm.

Estimates for each option:

1. (Increase timeout) 2 hours
2. (Split into more jobs) 4 hours
3. (Share HEALPix) 8 hours
4. (Optimize rectangular conversion) We will create a PR for the logging issue we found. Beyond that, we would need to do more investigation to provide a more informed estimate.

[pleasant-menlo]

we have purposely blocked files from being uploaded before the job has finished processing because this complicates the version determination and job deduping logic. I think the solution would be to increase the timeout of these jobs. @bryan-harter what do you think?

Could you provide more information about this blocking? Is it a policy of "don't upload anything until you're done", an error thrown on upload, a delay in processing uploads, etc.?

Given that these jobs are run on a daily cadence instead of being triggered by inputs, would the reasoning around early uploads change? Or would there still be problems for the spectral index maps that are directly downstream?

We are happy to increase the timeout or implement other solutions as well.

[jtniehof]

If option 3 (sharing the HEALPix output) will likely allow us to complete within the timeout, let's do that. It feels cleaner than more map subdivisions and buys us actual performance. Otherwise let's just split the jobs--basically I don't want to be in a position where we improve the calculation and still have to split the jobs.