Known Data Issues

Inconsistencies

September 21, 2018

Diagnosis

On 21 September 2018 at approximately 00:00UTC+00:00, the Stack Exchange API was unreachable by the data-aggregation utility. As the data-aggregation utility had not been a particularly high priority for me, it was not noticed for over two months, on 4 Dec 2018 at 00:39 UTC+00:00.

Remediation

The aggregator was run upon discovery of a failure to retrieve data.

Limitations

Data for 21 September 2018 through 3 December 2018 could not be gathered at the usual 00:00UTC+00:00, and was not gathered later. As a result, all data for this date range was unable to be collected.

Prevention

N/A

October 5, 2016

Diagnosis

On 5 October 2016 at approximately 00:00UTC+00:00, the Stack Exchange API threw an error on the badges/name endpoint which caused the Stack Exchange Statistics Explorer Data Aggregator to crash and fail to complete it's assigned duty. The API threw an Error 500: Internal Server Error code and the Data Aggregator was not prepared for that. As a result, data could not be collected at the usual 00:00UTC+00:00 time.

The exact request that created the first error was: https://api.stackexchange.com/2.2/badges/name?order=asc&sort=rank&site=stackoverflow&pagesize=100&page=1

This is documented on Stack Apps Bug Report 7064: http://stackapps.com/q/7064/.

Remediation

Upon discovery of the issue at approximately 06:00UTC+00:00, the aggregator was modified and re-run to collect data as soon as possible. This new API request was not sent until 06:11:03UTC+00:00.

Limitations

Data for 4 October 2016 could not be gathered at the usual 00:00UTC+00:00, and was not gathered later. As a result, all data for this date was unable to be collected, and measures are being taken to make sure that the issues created by this gap can be remedied.

Data for 5 October 2016 could not be gathered at the usual 00:00UTC+00:00, but was gathered later. As such any data gathered on this date may be slightly skewed in relation to 3 October 2016 and 6 October 2016. This will not affect data over a long-period, provided neither endpoint of that period is 5 October 2016, or 4 October 2016. This will affect larger sites much more than smaller ones. Badge stats for this date will not be acquired for any sites.

Prevention

This Data Aggregator has been rewritten to accomodate any failures on the badges/name endpoint, and will now simply skip adding badge stats for the site/day when this error occurs.