You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the NIST importing functionality is too slow, often taking many hours to import the dataset. Taking a look into the codebase it looks like where spawning multiple database transactions in order to import a single entry:
Since a lot of these entries are completely independent of each other we should batch insert them into the database in a single transaction (even packing 1000s of CVEs at a time).
Which will result in a single BEGIN/COMMIT per chunk rather than multiple per-CVE. The relational properties are still held within the transaction itself.
The text was updated successfully, but these errors were encountered:
Currently the NIST importing functionality is too slow, often taking many hours to import the dataset. Taking a look into the codebase it looks like where spawning multiple database transactions in order to import a single entry:
https://github.com/Exein-io/kepler/blob/558afe222b3c21c72a66d26ea1e93695d2c3751c/kepler/src/main.rs#L146-L187
Since a lot of these entries are completely independent of each other we should batch insert them into the database in a single transaction (even packing 1000s of CVEs at a time).
Which will result in a single
BEGIN/COMMIT
per chunk rather than multiple per-CVE. The relational properties are still held within the transaction itself.The text was updated successfully, but these errors were encountered: