Slam script incorrect ResultInterferometer and ResultImaging adapt_image calls

[AstroAaron]

Running the slam script source_pixelized.py on my own data, I found that line 77 in source_pix.py in slam/ produces the following error:

The issue may be that it calls the adapt_images from ResultInterferometer objects incorrectly. Looking into ResultInterferometer, I found that this class has the "adapt_images_from" function. Replacing any instance of "source_lp_results.last.adapt_images" in source_pix.py (and equivalent calls further down the pipeline!) with "source_lp_results.last.adapt_images_from(use_model_images=True)" appears to be a fix. However, I only see that the source_pix search is running and don't have results yet.

I tested if the adapt_image problem at least also appears for the imaging test data "simple__no_lens_light" and it produced the same error:

However, I don't have a workaround in this case.

[Jammy2211]

Thanks for the issue :).

I did a lot of updates to adapt_images in the past week, including a new release, and certain example scripts may not have been properly updated!

It sounds like you landed on the fix yourself, but post here if you have any other issues or the search which started appears to be stuck.

[AstroAaron]

Thank you for this fantastic tool ;)

The interferometer run eventually failed after 9 bounds, possibly due to an out-of-memory issue leading to a core dump, and no further report in its log file. Since I am working on a hpc, I cannot track exactly if this was the case.

Re-running the slam script should've started the process from where it left off, it did but immediately gave the following error:

This might be a consequence of some import issues from the previous run but is a bit too deep for me to understand where the problem comes from.

I renamed the output folder defined in the source_pix.py equivalent (I use a slightly modified version) in the output folder in my file system (not in the script) and am running the pipeline again. It appears to be running the search using the results of the parametric run.

[Jammy2211]

Yeah Im unsure on the error above, but it seems reasonable that the results folder had a few incorrect files output there and rerunning it from scrtch (by renaming a few folders) would sort it.

If errors persist I can do an end-to-end interferometer run myself and check that it all works smoothly.

[AstroAaron]

After a couple of weeks, the first of the two pixelation runs was finally completed. However, this time it resulted in the same error as before:

Which means it was not an incorrect file issue earlier in the post. This also means it should be a general issue, so I would appreciate it greatly if you could run the end-to-end interferometer run for yourself. I have not updated the code
since three weeks ago, but I couldn't find anything in the recent commits where I could see this issue fixed yet.
To me, it appears that some kind of plotting function cannot read or find the pixel_scales from the real_space_mask in the dataset, since we don't specify pixel_scales directly in the "dataset" object.

[Jammy2211]

I have pushed a fix to the main branch of PyAutoArray.

However, you may wish to edit your source code with the fix manually, just to make sure the code does not lose backwards compatibility with your old results (I don't think it would, but better safe than sorry).

The fix requires you to go to this script in your virtual environment:

https://github.com/Jammy2211/PyAutoArray/blob/main/autoarray/inversion/plot/inversion_plotters.py

Go to line 282:

            self.mat_plot_2d.plot_array(
                array=self.inversion.data,
                visuals_2d=self.get_visuals_2d_for_data(),
                auto_labels=AutoLabels(title=f" Data"),
            )

And replace it with the following code:

        try:
            self.mat_plot_2d.plot_array(
                array=self.inversion.data,
                visuals_2d=self.get_visuals_2d_for_data(),
                auto_labels=AutoLabels(title=f" Data"),
            )
        except AttributeError:
            pass

[AstroAaron]

Thank you for the fast fix! Works like a charm now.

[AstroAaron]

Hello again,

I have noticed that actually my pixel run started from scratch instead of continuing/finishing the run and producing all the required files, i.e. ".completed", "model.results", "search.summary", etc. So I think that, in the step of updating the final results, where it produced the error last time, it automatically removed the checkpoint.hdf5 file which enabled it to start again from there. Now it always wants to start from the beginning, even though the sampling is supposed to be finished. The other files, see screenshots, are still there, and also "samples_info.json" shows that it has all 26200 samples.

Is there a way to either: create the checkpoint hdf5 file from what I have from the json and csv files?
Or, a way to have the search somehow understand that it is finished already and only has to create the final files from what is there?

The files I have of the results:

[Jammy2211]

Can you try putting a .completed file in the unique identifier folder and running again?

Autolens checks if this file exists, and if it does it should bypass the search and load the result for doing stuff like visualization from samples.csv.

[AstroAaron]

That's a great idea! It works. Out of curiosity, should the second analysis for the second run in source_pix.py have a positions_likelihood from the parametric run as the first pixel run did? Speaking about interferometer data, of course.

[Jammy2211]

source_pix[2] does not need a positions_likelihood, because the mass model is fixed during this search (and we therefore know the positions threshold will always be met.

You are defo gonna wanna make sure your mass[1] search has a positions_likelihood though!

[AstroAaron]

Perfect, thank you! I did it correctly then. Fingers crossed that there are no further issues in this pixel run.
It failed the very same moment I wrote the above:

Afterward, I set raise_inversion_positions_likelihood_exception=False explicitly in the analysisInterferometer object used in the second search.

[Jammy2211]

Strange, are you able to input a positions_likelihood into source_pix[2] (something like analysis=analysis_1.positions_likelihood.

No idea why that exception is being raised, I'll note its a bug.

Alternatively, you can prob also disable it by changing the following setting in general.yaml:

test:
  disable_positions_lh_inversion_check: true

[AstroAaron]

Two ways worked for me. One is:

    analysis = al.AnalysisInterferometer(
        dataset=analysis.dataset,
        adapt_images=result_1.adapt_images_from(use_model_images=True),
        **raise_inversion_positions_likelihood_exception=False,**
        settings_inversion=al.SettingsInversion(
            image_mesh_min_mesh_pixels_per_pixel=3,
            image_mesh_min_mesh_number=5,
            image_mesh_adapt_background_percent_threshold=0.1,
            image_mesh_adapt_background_percent_check=0.8, use_linear_operators=True,
        ),
    )

And alternatively setting:
analysis_1 = al.AnalysisInterferometer( dataset=analysis.dataset, positions_likelihood=analysis.positions_likelihood, ...
I did not try changing the setting in general.yaml since it is working atm.

While it is running now, it is only using a single core and only spawns one thread to generate the initial samples of the source_pix[2] and that is taking forever, i.e. more than 14h (unknown how long it actually needs to take). It usually takes only ~2h to do the first 300 samples with nautilus. Is there a way of allowing multithreading or some sort of parallelization in the autofit.non_linear.initializerstep?

2024-03-01 11:25:51,288 - autofit.non_linear.initializer - INFO - Generating initial samples of model, which are subject to prior limits and other constraints.

I already have set the number_of_cores setting in non_linear.yaml to 16 (max. I can use). The same is defined in settings_search.

Edit: Nvm, now it did spawn the other threads. However, it is only using a single core. Before, I've only seen this when the multiprocessing didn't work. I tested DynestyStatic on my laptop and the multiprocessing works fine (for "Imaging" data), all 4 cores I set up were being used.

[Jammy2211]

Is there a way of allowing multithreading or some sort of parallelization in the autofit.non_linear.initializerstep?

Not currently, I think, I can raise an issue on PyAutoFit for us to implement this (its not often it really matters but clearly for your usecase it does lol).

I'm unsure why the threading isn't quite doing what you'd expect for source_pix[2]. Sounds like you're just about ok now, but I will keep an eye on it when I do my next round of profiling autolens.