Data cleaning report generates gaps on deleted iterations

[iperdomo]

When a user deletes an iteration of a repeatable question group, the DATA CLEANING report generates gaps in between. Example:

In the example the iteration=0 for form instance 579789116 and iteration=1 for instance 573639119 were deleted, but the sheet created for the repeatable question group (Group 3) contains gaps.

Sample dataset:

• Instance: UAT1
• Form Id: 571789118

[stellanl]

Found this TODO in org.waterforpeople.mapping.dataexport.InstanceData.InstanceData(SurveyInstanceDto, Map<Long, Map<Long, String>>):

        // Need to normalize the response map and add empty answers for missing iterations
        // as well as make sure that the iterations are sorted
        // TODO: Drop iterations with all empty answers (?).

So the "?" just became a "!"

We could renumber the iterations to have no index gaps. However, that should probably be done on data ingestion, not as part of report generation, as then the index will not match when importing after data cleaning.

[iperdomo]

We could renumber the iterations to have no index gaps. However, that should probably be done on data ingestion, not as part of report generation, as then the index will not match when importing after data cleaning.

Yeah, let's not mess with re-indexing answers. We're working on not having gaps in the app - akvo/akvo-flow-mobile#1564

[iperdomo]

I checked the same report on the most recent version of the deployed service. There are gaps in the indices, but not empty lines 👍

[iperdomo]

Code deployed https://travis-ci.org/akvo/akvo-flow-services/builds/655267535