Releases: PyAutoLabs/PyAutoLens
Winter 2022 Release
The primary new functionality are new source-plane pixelization (Delaunay triangulations and a Voronoi mesh) and regularization schemes which:
- Use interpolation when pairing source-pixels to traced image-pixels.
- Use a derivate evaluation scheme to derive the regularization.
These offer a general improvement to the quality of lens modeling using inversions and they correspond to the following classes:
Other features include:
-
Directly fitting a lens model to a lens model quantity (e.g. the deflection angles, convergence) as opposed to using data (https://github.com/Jammy2211/autolens_workspace/tree/release/scripts/misc/model_quantity).
-
Cored steep elliptical (CSE) implementation of various stellar and dark matter mass profiles for deflection calculation speed up (https://arxiv.org/pdf/2106.11464.pdf).
-
Simulating lens datasets where the source signal-to-noise ratio is an input (https://github.com/Jammy2211/autolens_workspace/blob/release/notebooks/imaging/simulators/misc/manual_signal_to_noise_ratio.ipynb).
2021.10.14.21
Note on backwards compatibility
The unique identifers of certain lens model will change as a result of this release, meaning that backwards compatibility may not be possible. We have a tool which updates the identifiers to this version such that existing results can be updated and retained, please contact me on SLACK if this is necessary.
Function Renames
Many core functions have been renamed for conciseness, for example:
deflections_2d_from_grid -> deflections_2d_from
convergence_2d_from_grid -> convergence_2d_from
This should not impact general use and the workspace has been updated with new templates using these functions.
Double Source Plane Lens Inversions
Reconstruction of multiple strong lensed sources at different redshifts (e.g. double Einstein ring systems) is now supported, including full model-fitting pipelines. The API for this is a natural extension of the existing API whereby multiple sources are allocated a Pixelization and Regularization:
lens = af.Model(
al.Galaxy,
redshift=0.5,
bulge=af.Model(al.lp.EllSersic),
mass=af.Model(al.mp.EllIsothermal)
)
source_0 = af.Model(
al.Galaxy,
redshift=1.0,
mass=al.mp.SphericalIsothermal,
pixelization=al.pix.VoronoiMagnification,
regularization=al.reg.Constant,
)
source_1 = af.Model(
al.Galaxy,
redshift=2.0,
pixelization=al.pix.VoronoiMagnification,
regularization=al.reg.Constant,
)
model = af.Collection(galaxies=af.Collection(lens=lens, source_0=source_0, source_1=source_1))
The following workspace examples demonstrate double source modeling and visualization further:
https://github.com/Jammy2211/autolens_workspace/blob/release/scripts/imaging/modeling/mass_total__source_sis_parametric__source_parametric.py
https://github.com/Jammy2211/autolens_workspace/blob/release/scripts/imaging/chaining/double_einstein_ring.py
https://github.com/Jammy2211/autolens_workspace/blob/release/scripts/imaging/chaining/pipelines/double_einstein_ring.py
https://github.com/Jammy2211/autolens_workspace/blob/release/scripts/plot/plotters/InversionPlotter.py
https://github.com/Jammy2211/autolens_workspace/blob/release/scripts/plot/plotters/FitImagingPlotter.py
Signal To Noise Light Profile Simulations
A class of signal-to-noise based light profiles, accessible via the command al.lp_snr, are now available. When used to simulate strtong lens imaging, these light profiles automatically adjust their intensity parameter based on the noise properties simulation to give the desired signal to noise ratio:
bulge=al.lp_snr.EllSersic(
signal_to_noise_ratio=50.0,
centre=(0.0, 0.0),
elliptical_comps=al.convert.elliptical_comps_from(axis_ratio=0.9, angle=45.0),
effective_radius=0.6,
sersic_index=3.0,
),
When combined with a Tracer the signal to noise of the light profile's image is adjusted based on the ray-traced image, thus it fully accounts for magnification when setting the signal to noise.
A full description of this feature can be found at this link:
W-Tilde Inversion Imaging Formalism
All Imaging Inversion analysis uses a new formalism for the linear algebra, which provides numerically equivalent results to the previous formalism (which is still implemented and used in certain scenarions).
The W-tilde formalism provides a > x3 speed up on high resolution imaging datasets. For example, for HST images with a pixel scale of 0.05" and a circular mask of 3.5", this formalism speeds up the overall run-time of a fit (e.g. one evaluation of the log likelihood function) from 4.8 seconds to 1.55 seconds. For higher resolution data or bigger masks an even more significant speed up is provided.
Users so not need to do anything to activate this formalism, it is now the default method used when an inversion is performed.
Implicit Preloading
Imaging and Interferometer analysis now use implicit preloading, whereby before a model-fit the model is inspected and preloadsare automatically generated for the parts aspects of the model-fit which do not change between each lens model. Previously, these would have been recomputed for every model fit, making the log likelihood evaluation time longer than necessary.
Example quantities which are stored via implicit preloading are:
- If the light profiles of all galaxies are fixed, their corresponding blurred image-plane image is preloaded and reused for every lens model fit.
- If the mass profiles of all galaxies are fixed, the deflection angles and ray-tracing do not change. Preloading is used to avoid repeated computation.
- Numerous aspects of the linear algebra of an inversion can be preloaded depending on which parts of the model do or do not vary.
This will provide significantl speed up for certain lens model fits.
2021.8.12.1
- Fixed installation issues due to requirement clashes with scipy.
- Database and aggregator support GridSearch model-fits, primarily for dark matter subhalo scanning models.
- Aggregator supports generation of tracers and fits which are drawn randomly from the PDF, for error estimation.
- Visualization of 1D light profile, mass profile and galaxy profile quantities with errors via the aggregator.
- More visualization tools, described at https://github.com/Jammy2211/autolens_workspace/tree/release/scripts/plot
Minor fix on pyquad import
Removed the use of pyquad from the EllipticalIsothermal mass profile's potential_2d_from_grid method.
Switch to Date Versioning Scheme
This release switches our versionin scheme to dates of the format year.month.day.release. This version is shared across all PyAuto projects.
There is no major new functionality in this release.
v1.15.3
This release brings in a number of features for improved model-fitting, all of which come from an updated to PyAutoFit:
- First class support for parallel Dynesty and Emcee model-fits. Previously, parallel fits were slow due to communication overheads, which are now handled correctly with PyAutoFit. One can expect a speed up close to the number of CPUs, for example a fit on 4 CPUs should take ~x4 less time to run. The API to perform a parallel fit is as follows:
search = af.DynestyStatic(
path_prefix=path.join("imaging", "modeling"),
name="mass[sie]_source[bulge]",
unique_tag=dataset_name,
nlive=50,
number_of_cores=1, # Number of cores controls parallelization
)
- In-built visualization tools for a non-linear search, using each non-linear search's inbuilt visualization libraries. Examples of each visualization are provided at the following link:
https://github.com/Jammy2211/autolens_workspace/tree/release/scripts/plot/search
-
Updated to the unique tag generation, which control the output model folder based on the model, search and name of the dataset.
-
Improved database tools for querying, including queries based on the name of the specific fit of the non-linear search and the dataset name unique tag.
v1.15.2
v1.15.1
MAJOR RELEASE:
This release is a major overhaul of the PyAutoLens lens modeling API, and as such means that all results are not backwards compatible. Do not update to this version of autolens if you need to use previous results!
The main purpose of this release are major changes to the PyAutoLens source code that utilize the ‘.sqlite’ database output feature. Large suits of lens modeling results can be output directly to a ‘sqlite’ database, such that the results of model-fits to many tens of thousands of lenses can be efficiently loaded and manipulated in a Jupyter notebook. For those who have used the Aggregator, the Aggregator now interacts with the database.
The database has allowed us to remove some core components and features of PyAutoLens, in particular the removal of phases, output-path tagging and Setup objects. The new API for these features is nevertheless easily recognisable for existing users, and the autolens_workspace contains numerous example scripts for the new API.
Over the next month, we will be fully testing the database feature and are likely to make a number of changes to it. Therefore, if you wish to use the database now you should speak to me on SLACK first.
Phase Removal:
Phases have been completed removing from PyAutoLens, meaning that the PhaseImaging object used to perform model-fitting no longer exists. The following code which would have been used to perform a model fit:
phase = al.PhaseImaging(
search=af.DynestyStatic(name="phase[example]",n_live_points=50),
galaxies=dict(lens=lens_galaxy_model, source=source_galaxy_model),
)
result = phase.run(dataset=imaging, mask=mask)
Is now performed as follows:
search = af.DynestyStatic(name="search[example]", nlive=50)
analysis = al.AnalysisImaging(dataset=imaging)
result = search.fit(model=model, analysis=analysis)
Tagging Removal + Unique Identifier:
The tagging of output folders, where their name was customized based on the model fitted and search used, has been removed. This has been replaced with a unique identifier string which forms the inner folder of the model-fit output path.
This makes the output paths of results consist of many less folders.
SQLite Database:
The database moves the results that are output from the output folder to a ‘.sqlite’ database file, which can then be access like the Aggregator. This database runs significantly faster than the aggregator and supports advanced queries. Information on the database can be found in the database folder on the workspace.
Pipelines + SLaM:
The removal of phases and use of a unique identifier has allowed us to completely change how pipelines runner scripts are written, in a significantly more concise and readable way. If you are familiar with pipelines, the new API should be instantly familiar, but allows for a lot more customization is how the model-fit is performed. An example of a SLAM runner is linked to below:
HowToLens:
I have improved the HowToLens lectures (chapter 1 in particular) and updated them for the new API. In particular, chapter 2 now focused on just lens modeling whereas chapter 3 focused entirely on search chaining + pipelines.
GalaxyModel Removal:
The GalaxyModel object has been removed, and can now be created via the af.Model command:
lens_galaxy_model = af.Model(al.Galaxy, redshift=0.5, mass=al.mp.SphIsothermal)
1D Data Structures + Figures:
One dimensional data structures (e.g. Array1D, Grid1D) have been added. These can be input into Profile objects to compute quantities like the convergence in 1D, e.g. convergence_1d_from_grid(grid=grid).
If a 2D gird is input into a 1D function, the 2D grid is projected to 1D from the centre of the profile and along its major axis, such that this quantity can be cleanly plotted in 1D.
One dimensional plots of the convergence and potential are now accessible in ProfilePlotter and GalaxyPlotter objects.
Name changes:
The following names of objects have been changed:
`Elliptical` -> `Ell` (e.g. `EllipticalIsothermal` -> `ElIIsothermal`)
`Sphericall` -> `Sph` (e.g. `SphericalIsothermal` -> `SphIsothermal`)
`CollectionPriorModel` -> `Collection`
`PriorModel` -> `Model`
`image_from_grid` -> `image_2d_from_grid`
`convergence_from_grid` -> `convergence_2d_from_grid`
`potential_from_grid` -> `potential_2d_from_grid`
`deflections_from_grid` -> `deflections_2d_from_grid`
Journal of Open Source Software Zenodo Release
Tagged release for PyAutoLens JOSS submission.
1.12.0: - Interferometer analysis via PyLops is now live!
API changes:
A number of data structure objects have been renamed to make their dimensionality explicit (e.g. Grid -> Grid2D , Array -> Array2D). The method in_2d of data structures is now called native whereas in_1d is now slim.
Visualization:
- Major visualization refactor with minor change to Plotter API.
- Methods which wrap matploitlib methods now have a dedicated config file for default settings.
- Visuals can be manually added to any figure, or plotted via Include. plotter methods are now Plotter classes instead of plotter functions.
- A complete API reference guide of the new visualization tools can be found at autolens_workspace/notebooks/plot. This includes examples of how to plot every object and how to customize every matplotlib method that is called.
Decomposed Lens Modeling pipelines:
Although decomposed (e.g. stellar + dark) modeling has been feasible in PyAutoLens for a while, the SLaM pipelines on the autolens_workspace have now been comprehensively tested on mock and real datasets and are ready for generic adoption to any science use-case.
See autolens_workspace/notebooks/advanced/slam/with_lens_light for example pipelines.
Interferometer:
Support for interferometer analysis with PyLops is now fully functionality, which makes analysing millions of visibilities with pixelized sources feasible.