This code has been working great for me for months but yesterday I asked it to pull a particularly large batch of observations (over 4000 so still not huge) and it started getting this error from the fromJSON for loop that calls the observations in pages of 200 each:
Error in parse_con(txt, bigint_as_char) : parse error: premature EOF
(right here) ------^
I get the impression from some search results that “premature EOF” indicates an issue with the connection to the server sending out the data? That would be consistent with the way the error behaves; sometimes it hits the error on page 4, sometimes page 16, sometimes page 20. I thought this might be an issue of calling too many times too quickly, but even putting in “Sys.sleep(300)” isn’t enough to keep the error from happening. In other words the data is fine and I did manage to get it all eventually, it just trips up at arbitrary points when I pull it all at once. Is there a way to make the function check if the page came through and if not, fetch it again until it does? Or some more fundamental way to address this issue?
This is the stopgap code I wrote to check for errors and retry if the pull failed. Not sure if this is problematic in some way or a reasonable way to handle this? It works though.
nobs <- fromJSON(url)
npages <- ceiling(nobs$total_results / 200)
xout <- flatten(nobs$results)
xout <- xout[,keep]
for(i in 2:npages) {
# Get json and flatten
page <- paste0("&page=", i)
while (is.null(x)){
x <- tryCatch(fromJSON(gsub("&page=1", page, url)),
error = function(e)
return(NULL))
Sys.sleep(1)}
x <- flatten(x$results)
x1 <- x[,keep]
xout <- rbind(xout,x1)
x <- NULL
}