v0.13.0

Release 0.13.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix null values in YAML-style model parameter schema files. (#1437)

New Features

• Improvements for single pe setting workflow including plotting and metadata handling. (#1398)
• Add an application that allows to extract array and shower data and to save them in hdf5 format for the later calculation of production configuration limits. (#1402)
• Add integration tests to production image generation. (#1420)
• Improve reading of single parameter and files from database. Add plotting of tabular data using parameter_version. (#1426)
• Improved plotting of tabular data; adding error bars and schema validation for plotting configuration. (#1429)
• Add function to get all model parameter data for all model versions. (#1432)

Maintenance

• Run unit and integration test on database setup in local test environment. (#1424)
• Remove tests from pypi package. Remove obsolete __init__.py files. (#1430)
• Change base python version from 3.11 to 3.12. (#1433)
• Update exporting of sim_telarray model to simtools taking into account the updated file naming (included parameter versions). (#1438)
• Robust getting of username in metadata collector in case no user is defined on system level. (#1442)

v0.12.0

Release 0.12.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

0.12.0 - 2025-03-11

Bugfixes

• Bugfix in database-upload related string to list conversion. (#1423)
• Fix setting of db_api_authentication_database through env variable. (#1425)

v0.11.0

Release 0.11.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

Bugfixes

• Fix yamllint options in pre-commit (was essentially switched off). Fix syntax in many YAML files. (#1404)
• Fix date writing to ensure RFC 3339-conform date format. (#1411)
• Fix reading of combined units from model parameter unit string. (#1415)

New Features

• Adding two new applications for generating model parameter documentation reports for a given array element either for a given production, or across multiple productions. (#1185)
• Add building of optimized corsika binaries (e.g., avx2, avx512) and building of images for different corsika versions. (#1355)
• Add a simtool to derive limits for energy, viewcone and radius for a simulation. (#1356)
• Add generic simtools-run-application to run one or several simtools using a single configuration file. (#1379)
• Add reading of database collections from model parameter schemas. (#1380)
• Add module for simtools dependency management. Allows to retrieve e.g. sim_telarray and CORSIKA versions. (#1383)
• Expanded functionality of application configuration workflows based on the integration test configurations. (#1389)
• Set user in docker images and add possibility to add user information from env/command line. (#1400)
• Allow to mark integration tests in their use case config with a corresponding CTAO use case or requirement ID. (#1405)
• Run check for changelog entry only on PRs ready for review. (#1416)

Maintenance

• Maintenance in production configuration removing not needed flags and only use metric files. (#1376)
• Rename derive-limits application to production-derive-limits. (#1378)
• Add YAML file with build opts into CORSIKA/sim_telarray images. (#1382)
• Add unit tests for version.py. (#1384)
• Improve function naming and unit test coverage in simtools.utils.names. (#1390)
• Change checks for software updates from weekly to monthly. (#1394)
• Remove duplication in converting sim_telarray string-lists (values, units) to python lists. (#1395)
• Improve reading of design models; refactoring and improved docstrings for simtools.utils.names. (#1396)

Simulation model

• Introduce MSTx-NectarCam and MSTx-FlashCam design models for MSTs (allow both sites). (#1362)

v0.10.0

Release 0.10.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

Bugfixes

• Schema validation bugfixes. (#1330)

New Features

• Add a simtool to derive (normalize) the single p.e. amplitude spectrum using the sim_telarray tool norm_spe. (#1299)
• Add validation of simtools-generated configuration files for sim_telarray with reference files. (#1322)
• Add schema for fake mirror list. (#1338)
• Add additional queries to db handler to retrieve model versions etc. (#1349)
• Add fixed database version for testing (instead of LATEST). (#1365)

Maintenance

• Major restructering of database routines plus introduction of new simulation model. (#1316)
• Improve testing of results and allow comparison of lists of floats in json/YAML files. (#1319)
• Remove database sandboxes . (#1336)

What's Changed

• Fix label setting in validate_camera_efficiency by @GernotMaier in #1320
• Update pre-commit hooks by @github-actions in #1325
• Clear default sim_telarray cfg directories from all calls by @orelgueta in #1324
• Allow for validation of simtools-generated configuration files for sim_telarray with reference files. by @GernotMaier in #1322
• Schema validation bug fixes. by @GernotMaier in #1330
• Reference model corrections (new db simulation model) by @GernotMaier in #1331
• Set trigger current limit to 20 uA by @GernotMaier in #1334
• Correct branch to main for simulation model upload by @GernotMaier in #1335
• Improve testing of results and allow comparision of lists of floats in json/yaml files by @GernotMaier in #1319
• Add schema for fake mirror list by @GernotMaier in #1338
• Remove sandboxes by @GernotMaier in #1336
• Simulation model refactoring with refactoring of db_handler. by @GernotMaier in #1310
• Update testeff command for SSTs/SCTs to use fake mirror list by @GernotMaier in #1339
• Update pre-commit hooks by @github-actions in #1341
• Replace model parameter default schema versions by value read from model_parameter.metaschema.yml by @GernotMaier in #1332
• Remove duplication for schema validation plus some simplification by @GernotMaier in #1347
• Minor code / maintenance updates to production configuration setup. by @GernotMaier in #1345
• Production configuration schema and validation by @GernotMaier in #1348
• Add additional queries to db handler to retrieve model versions etc. by @GernotMaier in #1349
• Maintenance PR for simtel io histogram(s) modules. by @GernotMaier in #1352
• Update pre-commit hooks by @github-actions in #1358
• Schema typos and URL fixes by @GernotMaier in #1359
• Missing sphinx documentation by @GernotMaier in #1363
• Simtools for single pe derivation by @GernotMaier in #1299
• Add fixed database version for testing by @GernotMaier in #1365
• Add input validation for single pe tool. by @GernotMaier in #1350
• Add bin edges to hdf5 by @tobiaskleiner in #1354
• Make sure that Changelog entries are not forgotten. by @GernotMaier in #1368
• Update pre-commit hooks by @github-actions in #1369
• Validate meta_schema_url by @GernotMaier in #1366
• Prepare simtools release 0.10.0 by @GernotMaier in #1370

Full Changelog: v0.9.0...v0.10.0

v0.9.0

Release 0.9.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

Bugfixes

• Add correct handling of return codes from simulation software to prevent reports of success even though a job failed (simulate_prod) (#1289)
• Fix setting of activity:end time in the metadata collector. (#1291)

New Features

• Add an application to plot tables from a file (from file system or for a model parameter file downloaded from the DB). (#1267)
• Enhancements to file testing within integration tests. (#1279)
• Add a tool to prepare job submission with simulate_prod for a HTCondor system. (#1290)
• Remove functionality in db_handler to read from simulation model repository. (#1306)

Maintenance

• Change dependency from ctapipe to ctapipe-base for ctapipe v0.23 (#1247)
• Remove pytest dependent from user installation. (#1286)

What's Changed

• Change in optimization algorithm for the derivation of the random reflection angle parameter. by @GernotMaier in #1275
• Use astropy-base package in conda environment. by @GernotMaier in #1272
• CTAO CR spectra package now available from conda by @GernotMaier in #1277
• Update pre-commit hooks by @github-actions in #1278
• Update pre-commit hooks by @github-actions in #1282
• Improve db configuration checks by @GernotMaier in #1281
• Replace os.system by subprocess.run and Pathlib commands. by @GernotMaier in #1283
• Improvements for the job submission using HT Condor by @GernotMaier in #1279
• Move test_commandline_tools as test after simtools-prod container generation. by @GernotMaier in #1285
• Add a simple tool to prepare job submission with simulate_prod for a HTCondor system. by @GernotMaier in #1290
• Set correct activity:end in metadata by @GernotMaier in #1291
• Handle return codes correctly for simulation productions by @GernotMaier in #1289
• Add corsika log file collection for --pack_for_grid_register in simtools-simulate-prod by @GernotMaier in #1294
• Bump SonarSource/sonarqube-scan-action from 4.1.0 to 4.2.1 by @dependabot in #1296
• Update pre-commit hooks by @github-actions in #1295
• Fix failing unit tests in job_manager by @GernotMaier in #1297
• Update pre-commit hooks by @github-actions in #1298
• Remove pytest dependent from user installation. by @GernotMaier in #1286
• Optimize scipy imports in camera module by @GernotMaier in #1292
• Use-friendly naming of application documentation by @GernotMaier in #1300
• Fix validation of model parameters using schemas. by @GernotMaier in #1302
• Update pre-commit hooks by @github-actions in #1303
• Remove functionality in db_handler to read from simulation model repository by @GernotMaier in #1306
• Remove obsolute functions from db_handler by @GernotMaier in #1307
• Simplify db_handler.add_new_parameter and add parameter dict validation before writing by @GernotMaier in #1308
• Improvements to handle different versions of schemas. by @GernotMaier in #1309
• Update pre-commit hooks by @github-actions in #1311
• Simple plotting application for tabular data by @GernotMaier in #1267
• Preparation of data validation routines for db restructering by @GernotMaier in #1312
• Require explicit selection of model_version as command line parameter by @GernotMaier in #1313
• Fix site in integration test by @GernotMaier in #1314

Full Changelog: v0.8.2...v0.9.0

v0.8.2

Simulation tools and applications for the CTA Observatory.

Release 0.8.2 is a maintenance release with no changes in functionality. Main changes a new layout for the simtools code repository (src-layout).

What's Changed

• Update README.md by @tobiaskleiner in #1258
• type hints for models by @tobiaskleiner in #1253
• Update pre-commit hooks by @github-actions in #1262
• Improve help and printout for --telescope_model_file command line option. by @GernotMaier in #1263
• Bump SonarSource/sonarqube-scan-action from 4.0.0 to 4.1.0 by @dependabot in #1265
• Change simtools package layout from 'flat' to 'src'. by @GernotMaier in #1264
• Change schema url to scr-layout by @GernotMaier in #1268
• Changes related to src-layout move. by @GernotMaier in #1270

Full Changelog: v0.8.1...v0.8.2

v0.8.1

Simulation tools and applications for the CTA Observatory.

This is a bug fix release (0.8.1) to the the November 2024 monthly developer release v0.8.0. For major changes, see the CHANGELOG.md file.

What's Changed

• Fix pypi deployment by @GernotMaier in #1255
• Add deployment environment for pypi deploy by @GernotMaier in #1256

Full Changelog: v0.8.0...v0.8.1

simtools - release 0.8.0

Simulation tools and applications for the CTA Observatory.

This is the November 2024 monthly developer release v0.8.0. For major changes, see the CHANGELOG.md file.

What's Changed

• Improve tests for building and command line applications by @GernotMaier in #1062
• Bug fix on warning issued in get_version in db_handler by @GernotMaier in #1063
• Select latest version of model parameter databases by @GernotMaier in #1065
• Incidence angle, not incident angle by @GernotMaier in #1064
• Renaming to fix test issues by @GernotMaier in #1074
• Add CORSIKA parameters to DB and add reading code by @GernotMaier in #1056
• Address issues with pypi deployments by @GernotMaier in #1061
• Allow xfail for derived db in tests by @GernotMaier in #1071
• Documentation improvements by @GernotMaier in #1066
• Remove db backup CI by @GernotMaier in #1077
• Remove duplication of _validate_and_convert_value function. by @GernotMaier in #1076
• Flexible command line by @GernotMaier in #1073
• Add for the first-time a changelog by @GernotMaier in #1079
• Warnings for plt.plot(); connection to display server by @GernotMaier in #1078
• Simplify code for validate_camera_efficiency by @GernotMaier in #1075
• Improve unit tests for model_parameters; add read parameters from file by @GernotMaier in #1081
• Improve unit test execution time by @GernotMaier in #1082
• Bug fixes related to pytest-raises-with-multiple-statements (PT012) by @GernotMaier in #1085
• Limit git clone to depth 1 by @GernotMaier in #1088
• Addresses errors of type "pytest-raises-too-broad" from ruff by @GernotMaier in #1086
• Use array_element instead of telescope in names, model_parameter, db_handler by @GernotMaier in #1095
• Illuminator coordinates by @tobiaskleiner in #1091
• Remove outdated telescope naming by @GernotMaier in #1089
• Bump SonarSource/sonarqube-scan-action from 2 to 3 by @dependabot in #1101
• Bump super-linter/super-linter from 6 to 7 by @dependabot in #1100
• Improve execution time of integration tests. by @GernotMaier in #1092
• Remove obsolete db development scripts by @GernotMaier in #1102
• Wavelength should go up to 1000 nm by @orelgueta in #1114
• Require semantic version for model parameter version string by @GernotMaier in #1103
• Add procps to docker images by @GernotMaier in #1110
• Improve comments on pylint/astro.units by @GernotMaier in #1111
• Change y title for NSB camera efficiency by @orelgueta in #1117
• Improve speed of validate file using schema by @orelgueta in #1118
• Use run_number property by @orelgueta in #1116
• Generalize scripts to upload a model parameter repo to a new DB. by @GernotMaier in #1109
• Improve import efficiency of general.py by @GernotMaier in #1119
• validate_file_using_schema.py accepts a directory by @orelgueta in #1120
• Bug fix in reading model versions by @GernotMaier in #1122
• Ignore pre-releases of the model parameter database by @GernotMaier in #1121
• Move to strictly using semantic versions for models (remove e.g., 'prod5', 'prod6', 'Latest', 'Released') by @GernotMaier in #1123
• Improve primary particle API by @GernotMaier in #1127
• Fix triggering of telescopes in light emission package by @tobiaskleiner in #1128
• Test for two new lines at end of sim_telarray configuration file writing by @GernotMaier in #1135
• Introduce simtools testing modules; move tests from integration tests into this module by @GernotMaier in #1138
• Add EXPECTED_OUTPUT field to schema for integration test configuration by @GernotMaier in #1140
• Pre-commit update (and ruff update to 0.6.6) by @GernotMaier in #1144
• Check output of simulations by @orelgueta in #1139
• Missing documentation for new testing module by @GernotMaier in #1141
• Improve robustness of database scripts by @GernotMaier in #1143
• Update pyproject.toml for new pylint option max-positional-arguments by @GernotMaier in #1151
• Update sonar version in CI-unittests.yml by @GernotMaier in #1150
• Remove sim telarray native configuration files by @orelgueta in #1149
• Address numpy warnings in psf_analysis; fix minor bug by @GernotMaier in #1148
• Fix issue with config directory of illuminator simulations by @orelgueta in #1160
• Add nsb reference spectrum schema by @orelgueta in #1159
• Add Eshita as author by @EshitaJoshi in #1162
• Change affiliation to CTAO by @orelgueta in #1164
• Remove option for uploading of metadata to DB by @GernotMaier in #1166
• Run integration tests for indicated model version when no `model_vers… by @GernotMaier in #1163
• Update version number for corsika/sim_telarray package (20240913) by @GernotMaier in #1158
• Change null string to None for unit validation by @GernotMaier in #1169
• Add schema for correction of NSB spectrum to the telescope altitude by @orelgueta in #1171
• Allow for class Telescope in model parameters by @GernotMaier in #1173
• Fix testeff after removing native sim_telarray cfg by @orelgueta in #1154
• Run upload_from_model_repo from within the container by @orelgueta in #1134
• Update corsika/sim_telarray version to 240927 by @GernotMaier in #1175
• Fix testeff again by @orelgueta in #1177
• Bump SonarSource/sonarqube-scan-action from 2.3.0 to 3.0.0 by @dependabot in #1178
• Add design model schema by @GernotMaier in #1172
• Correct naming of design_model.schema.yml by @GernotMaier in #1179
• Check suffix of files in schemas/model_parameters by @GernotMaier in #1180
• Sort explicitely DB queries to retrieve newest entry by @GernotMaier in #1182
• Remove 3x duplication of string fs.files (reported as code smell by sonar) by @GernotMaier in #1170
• Improved array elements treatment by @GernotMaier in #1181
• Bump mamba-org/setup-micromamba from 1 to 2 by @dependabot in #1184
• Bug fix for reading telescope model for design and telescope ID. by @GernotMaier in #1187
• Integration test telescope position by @GernotMaier in #1188
• Simplify code to get simtel_configuration parameters from DB and improve call efficiency by @GernotMaier in #1183
• Add format_checkers module for json schema; explicit checking of array elements by @GernotMaier in #1191
• Missing API documentation; add check that this is not missed in future by @GernotMaier in #1194
• Remove old style configuration of ray-tracing tools. by @GernotMaier in #1142
• New array trigger definition by @GernotMaier in #1192
• Array triggers write simconfig by @GernotMaier in #1198
• Refactor validate_data_dict an...

simtools - prototype release v0.7.0

Prototype implementation of tools for the Simulation System of the CTA Observatory.

This is an intermediate development release with

• significant refactoring in the handling and reading of parameters from the DB.
• addition of application using the light emission package
• addition of application for trigger rate calculation

What's Changed

• Bump codecov/codecov-action from 3 to 4 by @dependabot in #800
• Add 2024, Feb 5 simtel version by @GernotMaier in #803
• Add CI for daily backups of the simulation model DB by @GernotMaier in #805
• Performance improvements by @GernotMaier in #810
• Failing linters by @GernotMaier in #809
• Add application to add a new parameter to the database from a json file by @GernotMaier in #807
• Fix failing raytracing unit test by @GernotMaier in #819
• Add pytest profiling to developer enviroment by @GernotMaier in #815
• fix environment name by @GernotMaier in #821
• Unit tests profiling by @GernotMaier in #822
• Unit vs units change in data metaschema by @GernotMaier in #814
• New simulation model by @GernotMaier in #761
• Address conda warning about pip by @GernotMaier in #831
• Re enable camera efficiency and ray tracing tests; files from test-data by @GernotMaier in #827
• Add pytest repeat to find dependencies between unit tests by @GernotMaier in #833
• Add -n auto options to pytest CI by @GernotMaier in #832
• Fixes broken container generation for new simulation model by @GernotMaier in #826
• Add application to import model parameters from simtel configuration file by @GernotMaier in #824
• Updated array elements naming and re-organisation of simulation model by @GernotMaier in #830
• Add readme hint for cloning of model_parameters repository by @GernotMaier in #840
• Allow to push packages for manual workflow by @GernotMaier in #838
• Improvements to model parameter schema validation by @GernotMaier in #836
• Add pip upgrade to prod docker file by @GernotMaier in #841
• Simplify file naming in names.py by @GernotMaier in #835
• Updates to get correct site and default values into DB by @GernotMaier in #842
• Update of to simtel / corsika package with date 240318 by @GernotMaier in #844
• Simplify application to add values to DB from json file; move to main application directory by @GernotMaier in #843
• Changes required for adding prod6 model to database by @GernotMaier in #847
• No need of test-data DB, as files are now in tests/resources by @GernotMaier in #856
• add Calibration by @tobiaskleiner in #859
• Add all files in a directory to database by @GernotMaier in #854
• fix boost-histogram by @VictorBarbosaMartins in #849
• Add codespell to pre-commit and apply it to files by @GernotMaier in #855
• add illuminator to instrument by @tobiaskleiner in #862
• Remove python 3.9 and add 3.12 from CI workflow by @GernotMaier in #860
• Use astropy units in jsonschema validation by @GernotMaier in #864
• Add validation of parameter name to be consistent with file name for for model_parameter files by @GernotMaier in #865
• Address sluggish startup of tools - replace astropy.io.misc.yaml by plain yaml by @GernotMaier in #813
• Units with spaces by @GernotMaier in #866
• Fix url to unit test by @GernotMaier in #867
• Simple changes to accelerate integration tests. by @GernotMaier in #868
• allow to add to calibration_devices by @tobiaskleiner in #870
• drop sandboxes by @tobiaskleiner in #872
• Reduce number of calls to DB pass plus some maintenance in db_handler by @GernotMaier in #869
• Generate docker files for different corsika/sim_telarray versions by @GernotMaier in #876
• Add unix find to development docker by @GernotMaier in #884
• Allow model_version command line configuration of integration tests by @GernotMaier in #878
• Add an application to upload all files from a model parameter repo into a new data base by @GernotMaier in #885
• Bug fix: drop metadata collection after tests by @GernotMaier in #888
• update docker documentation by @VictorBarbosaMartins in #891
• Re-enable docker images for arm64 architectures by @GernotMaier in #877
• Unify array elements in names.py and json schema by @GernotMaier in #889
• Name of simulation model database as configuration entry by @GernotMaier in #893
• Move model parameter schemas from gitlab into simtools by @GernotMaier in #894
• CI schema validation by @GernotMaier in #897
• remove if statement -> return testing all the output files in integra… by @VictorBarbosaMartins in #887
• Housekeeping: pylint and location of schema files by @GernotMaier in #898
• Integrate and use model parameter schema to determine sim_telarray and CORSIKA parameters by @GernotMaier in #895
• python 311 in pyproject.tom by @GernotMaier in #904
• Add a simtools-simulate-prod integration test for the South site by @GernotMaier in #883
• Set schema paths using importlib by @GernotMaier in #905
• Documentation maintenance updates by @GernotMaier in #900
• More integration tests by @VictorBarbosaMartins in #906
• Simulate prod test by @VictorBarbosaMartins in #875
• Improve import efficiency for UnitsErrors by @GernotMaier in #909
• Separate plot_camera from camera module by @GernotMaier in #907
• Ci pylint fixes by @tobiaskleiner in #914
• Integrate light emission by @tobiaskleiner in #802
• Allow to trigger CI (especially tests) by hand. by @GernotMaier in #920
• Run pre-commit as CI by @GernotMaier in #913
• Add metaschema and tests for integration tests config files by @GernotMaier in #922
• Unit tests in random order to make sure that unit tests are independent of each other by @GernotMaier in #921
• Trigger rate module and application by @VictorBarbosaMartins in #801
• Add telescope positions to database and improve ArrayModel by @GernotMaier in #910
• Add docker readme to general documentation by @GernotMaier in #930
• Trigger rate stacked files by @VictorBarbosaMartins in #935
• Read site names from list of array elements by @GernotMaier in #928
• Read simulation model versions from DB by @GernotMaier in #927
• Add pre-commit hook to avoid large files updates by @GernotMaier in #939
• Step 1 of ruff integration: auto fixes by @GernotMaier in #938
• Step 2 of ruff integration: fix issues by @GernotMaier in #941
• Step 3 of ruff integration: add ruff to CI (plus cleanup of different linter stages) by @GernotMaier in #943
• Add a section on model parameters to the documentation by @GernotMaier in #932
• Documentation swit...

simtools - prototype release v0.6.1

Prototype implementation of tools for the Simulation System of the CTA Observatory.

This is an intermediate development release which includes:

• bug fix to package generation for releases
• improvement to names.py to allow to use CTAO array elements IDs for telescope names.

Full Changelog: v0.6.0...v0.6.1