Feat: New anta_workflow plugin using PyAVD

[carl-baillargeon]

Change Summary

• anta_workflow Ansible action plugin
• anta_runner Ansible role
• get_device_anta_catalog function in PyAVD to generate the ANTA catalogs
• ANTA test index with their input factories in PyAVD

Requirements

NEEDS ANTA v1.3.0

On ANTA:

• feat(anta): Add Markdown report option to ANTA anta#740
• feat(anta.tests): Add VerifyBGPPeerSession + minor fixes on other BGP tests  anta#987
• fix(anta): Fix various issues in CSV and Markdown reporters anta#990

On AVD:

• Feat(eos_designs): Added rack, pod, dc, fabric information in the structured_config metadata #4827

Component(s) name

PyAVD

Proposed changes

Example playbook for anta_runner role:

---
- name: Validate AVD
  hosts: GLOBAL
  tasks:
    - name: Validate states on EOS devices
      import_role:
        name: arista.avd.anta_runner
      vars:
        
        # AVD catalogs generation settings
        avd_catalogs_enabled: true  # Enable/disable AVD catalogs generation
        avd_catalogs_dir: "{{ inventory_dir }}/anta/avd_catalogs"  # Directory to store AVD catalogs
        avd_catalogs_allow_bgp_vrfs: true  # Allow BGP VRFs tests in AVD catalogs
        avd_catalogs_filters:
          # Skip VerifyNTP for all fabric devices
          - device_list: "{{ groups['GLOBAL'] }}"
            skip_tests:
              - VerifyNTP
          # Only VerifyReachability will run on DC1 devices
          - device_list: "{{ groups['DC1'] }}"
            run_tests:
              - VerifyReachability
        structured_dir: "{{ inventory_dir }}/intended/structured_configs"  # Directory to store structured configuration files
        avd_structured_config_file_format: "yml"  # Format of structured configuration files
        
        # User-defined catalogs settings
        user_catalogs_dir: "{{ inventory_dir }}/anta/user_catalogs"  # Directory to store user-defined catalogs

        # Runner settings
        anta_runner_timeout: 30 # Timeout for eAPI calls
        anta_runner_batch_size: 10  # Number of devices to process in parallel
        anta_runner_tags: []  # Tags to filter tests, can be used in conjunction with `anta_tags` host variable to filter tests
        anta_runner_dry_run: false  # Dry-run mode
        anta_runner_logs_dir: "{{ inventory_dir }}/anta/logs"  # Directory to store debug logs when running `-vvv`
        
        # Report settings
        anta_report_csv_path: "{{ inventory_dir }}/anta/reports/anta_report.csv"  # Path to store the CSV report
        anta_report_md_path: "{{ inventory_dir }}/anta/reports/anta_report.md"  # Path to store the Markdown report
        anta_report_json_path: "{{ inventory_dir }}/anta/reports/anta_report.json"  # Path to store the JSON report
        anta_report_hide_statuses:  # Filter out success and skipped tests from the report
          - success
          - skipped
    

How to test

1- Follow theTo test this pull request snippet from github-actions to install the requirements
2- By default, you can simply run the following playbook:

---
- name: Validate AVD
  hosts: GLOBAL
  tasks:
    - name: Validate states on EOS devices
      import_role:
        name: arista.avd.anta_runner

3- Test different variables/behaviors following the example playbook in Proposed changes

To-do list

• Add reporting using anta.reporter.csv_reporter and anta.reporter.md_reporter. We should also be able to filter statuses (from plugin arguments) using ResultManager.filter()
• Input factory for VerifyAVTRole 4e20dbf
• Input factory for VerifyMlagStatus
• Input factory for VerifyRoutingProtocolModel 4e20dbf
• Input factory for VerifyAPIHttpsSSL
• Input factory for VerifySpecificIPSecConn 4e20dbf
• Input factory for VerifyStunClient 4e20dbf
• Input factory for VerifyNTP 4e20dbf
• Input factory for VerifyReloadCause 4e20dbf
• VerifyLLDP factory: Add support for validate_state: false configuration 4e20dbf
• Create the Ansible role (anta_runner) that will run the action plugin anta_workflows - Similar to cv_deploy & cv_workflows 18d98be
• Complete anta_workflow plugin documentation under plugins.modules
• Complete README.md for anta_runner role
• Add support for BGP VRFs (see Feat(eos_validate_state): Add validation for BGP IPv4 peers in VRFs #4538)
• Molecule tests

Some tests don't require any inputs, in that case we just update AVD_TEST_INDEX

Phase 2:

• Unit tests
• Expose more variables in the structured config schema for testing (accepted_pwr_supply_states, accepted_fan_states, accepted_xcvr_manufacturers, eos_version, etc.)
• Add more ANTA tests 🥇
• Improve reporting (HTML) when feat(anta): add test atomic results anta#937 is merged

[github-actions]

Review docs on Read the Docs

To test this pull request:

# Create virtual environment for this testing below the current directory
python -m venv test-avd-pr-4196
# Activate the virtual environment
source test-avd-pr-4196/bin/activate
# Install all requirements including PyAVD
pip install "pyavd[ansible] @ git+https://github.com/carl-baillargeon/avd.git@feat/move_anta_to_pyavd#subdirectory=python-avd" --force
# Install Ansible collection
ansible-galaxy collection install git+https://github.com/carl-baillargeon/avd.git#/ansible_collections/arista/avd/,feat/move_anta_to_pyavd --force
# Optional: Install AVD examples
cd test-avd-pr-4196
ansible-playbook arista.avd.install_examples

[sonarqubecloud]

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[ankudinov]

I've tested the PR on the following inventory and AVD dev image.

In arista.avd.eos_validate_state we used to have save_catalog. I never used it, but I find it a bit confusing that anta_runner combines that under avd_catalogs_enabled togerther with activating/deactivating default tests. That could be a good thing, but when combined with re-writing the whole anta/ directory content. After playing with avd_catalogs_enabled and avd_catalogs_dir a bit it’s easy to create quite confusing setup, that will never be auto-cleaned. I’d propose following:

• Overwrite everything, except user_catalogs_dir on every run. Also clean any old dirs and files.
• Dump avd catalog only when avd_catalogs_enabled: true (already the case)
• Introduce a knob to dump the combined catalog in some different location for debugging.

AVD catalogs are JSON by default. Do we have an option to dump them as YAML?

avd_catalogs_filters - I was able to make them work for hostnames, but not for groups. I'd consider at least adding some documentation. Ideally I'd convert device_list to a list, not a string, that accepts hostnames, group names and fails if host or group is not known.

anta_report_filter_statuses: [success] has no effect.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[carl-baillargeon]

I've tested the PR on the following inventory and AVD dev image.

In arista.avd.eos_validate_state we used to have save_catalog. I never used it, but I find it a bit confusing that anta_runner combines that under avd_catalogs_enabled togerther with activating/deactivating default tests. That could be a good thing, but when combined with re-writing the whole anta/ directory content. After playing with avd_catalogs_enabled and avd_catalogs_dir a bit it’s easy to create quite confusing setup, that will never be auto-cleaned. I’d propose following:

• Overwrite everything, except user_catalogs_dir on every run. Also clean any old dirs and files.
• Dump avd catalog only when avd_catalogs_enabled: true (already the case)
• Introduce a knob to dump the combined catalog in some different location for debugging.

AVD catalogs are JSON by default. Do we have an option to dump them as YAML?

avd_catalogs_filters - I was able to make them work for hostnames, but not for groups. I'd consider at least adding some documentation. Ideally I'd convert device_list to a list, not a string, that accepts hostnames, group names and fails if host or group is not known.

anta_report_filter_statuses: [success] has no effect.

Thanks for the review Petr! As discussed, we will implement the following feature in future PRs:
1- Introduce a knob to dump the combined catalog
2- AVD catalogs in YAML
3- Validate host + groups in the filters

[ankudinov]

LGTM

[gmuloc]

we can ignore coverage here - we are not having coverage on the ansible plugins right now but are looking at it - so we will take the hit.

[sonarqubecloud]

Quality Gate failed

Failed conditions
3.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[gmuloc]

I have not reviewed the code thoroughly but this is preview so we will need another round of review.

Lets expose this to testers in the field to get feedback