Add meta data column to data exports showing form version if available

[janagombitova]

Context

A survey form can have many versions resulting in the situation that a data set can consist of submissions made to the form but to different versions of the form. Sometimes this can result in the data looking a bit different (different options available in v1 and v2 of the form; or a question was first free text and now is cascade). When looking at the data Petra does not know from with version of the form the submissions are from, so it is harder for her to know why the data looks the way it does.

With this issue: #2707 we decided to collect the form version with the submissions.

Opportunity

See where we can show the form version with the submissions. One important place is in the data exports, as that is the most common way of working and handling the data. Thus the idea is to add another column to the meta data of each export with the Form version (if available). For data cleaning this column should not be editable.

To consider

We need to see how to name to column, what to do it there is no data. Will this change have any affects for Lumen?

With another iteration we should also consider where else to show the form version in the UI when viewing the submissions?

[stellanl]

I initially named the new column "Form version". Thoughts?
And I leave the cell empty if there is no data for it. Another approach is to put in a 0.

[janagombitova]

@stellanl I think that is a good approach for now till we get feedback from users

[stellanl]

You say above, "For data cleaning this column should not be editable." which is ok until we consider
what to do when data is being added through data cleaning. Set version to what is in the sheet? To 0.0? The latest published version?

[janagombitova]

With @stellanl we agreed to handle at data cleaning import the forms version the same way as we handle submission date and duration, for now. Submission date and duration are used by partners to check the quality of the data and its reliability and the form version falls under the same category, so we will handle it in the same way.

[stellanl]

One final question. Since the metadata column headers will now be used to find the content, should we ignore case? E.g. should "Device identifier", "Device Identifier", "device identifier" and "dEvIcE IDenTIfiER" all be valid?

[janagombitova]

That sounds reasonable enough to me. 👍

[stellanl]

It should avoid some support issues, and not harm anything, so I am doing it.

[stellanl]

Test plan:

1. Export data from a form, in all 3 formats. All should have a "Form version" column.
2. Edit the DATA_CLEANING spreadsheet:

• Change case of at least one metadata column.
• Change the form version for an existing form instance.
• Add a new instance (=Instance column empty) with a (numeric) form version.

3. Import the file.
4. In messages, check that the import ran successfully.
5. Export the form data again.
6. Check that the version of old form instance did not change, and that the new one has one.

[stellanl]

Documentation changes:

On the page "Data cleaning"

1. Add "Form version" to the list of columns that should not be modified.
2. Insert text about the column headers; how they must not be modified. (Actually, as long as they are correctly spelled, they can now be moved around, but it is probably just confusing to mention this...)

On the page "Importing in new data via Data cleaning" there a few issues:

1. The text "In the Identifier column add unique IDs to each row, for example: new-1; new-2; new-3. The IDs cannot be duplicated. " is not strong enough. In fact, these unique IDs must begin with "new-"
2. "importing in" should be just "importing".
3. "meta data" should be "metadata". https://en.wiktionary.org/wiki/meta_data