Rxjava 3 + Retrofit2 - Multiple Inserts To Db Problem

I am trying to do the following; sync a cloud DB using Retrofit to a local SqLite DB (Room) on a device. The DB could get large, around 100,000 registers or more, so the sync proce

Solution 1:

You posted a json response here before the edit.

CurrentPage:1,PageCount:65566,PageSize:1,RecordCount:65566

If I understand correctly, then you have 65k items and 1 item in each page. Meaning 65k pages which means 65k network calls. That's a lot. You could improve this design first.

1. Divide the entire records into a few pages (maybe even 10 or 20). 1 page will still have thousands of items if the entire records has 10s of thousands of items.
2. Then use gzip compression to compress the json responses for each page and serve that from the server. Or don't divide the records into pages and pass them all in one response compressed with gzip (if it's not that big).
3. Unzip the response on android, parse it and then do whatever you want.

This way you reduce a lot of network calls and possibly reduce the wait time for sync.

As to your actual rx question:

val pageSize = 100
viewModel.getRecordsCount()
    .map {
        // logic from `HandleResults` function// do some calculationvar numPages: Int = it.records / pageSize
        if (it.records < pageSize || it.records % pageSize != 0) {
            numPages++
        }
        return@map numPages
    }
    .flatMap { pages -> Observable.range(1, pages) }
    .flatMap { page -> viewModel.getItemsPerPage(pageSize, page) }
    .flatMap { itemSyncDetails ->
        val items = viewModel.insertAllLocal(itemSyncDetails.getItemList())
        return@flatMap Observable.just(items.size)
    }
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(....)

I notice that not 100% of all the records are inserted, every time that I run the process, the number of records inserted change, it is around 80% - 98%, but never 100%, even though all the Retrofit calls are sent.

Log the error in handleError function and see what the actual problem is.