Uploaded image for project: 'Realm Core'
  1. Realm Core
  2. RCORE-2193

download progress estimate is always 1.0

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None

      Expected results

      non-zero values reported during sync

      Actual Results

      estimate is always 1.0

      Steps & Code to Reproduce

      have some data on the server, open new realm and set the query to fetch some of it, wait for subscriptions to bootstrap, check reported values
      <details><summary>sample testcase to see the problem:</summary>

      TEST_CASE("progress_2") {
          Schema schema{ObjectSchema("TopLevel", {{"_id", PropertyType::ObjectId, Property::IsPrimary{true}},
                                                  {"queryable_int_field", PropertyType::Int},
                                                  {"payload", PropertyType::String}})};
          FLXSyncTestHarness harness("flx_download_progress", {schema, {"queryable_int_field", "payload"}});
      
          int num_objects = 9;
          auto add_objects = [&](SharedRealm& realm, const std::string& table_name, int num_objects) {
              auto make_one = [](int64_t idx) {
                  return AnyDict{
                      {"_id", ObjectId::gen()}, {"queryable_int_field", idx}, {"payload", random_string(1024 * 1024)}};
              };
      
              CppContext c(realm);
              for (int i = 0; i < num_objects; ++i) {
                  realm->begin_transaction();
                  Object::create(c, realm, table_name, std::any(make_one(i)));
                  realm->commit_transaction();
              }
          };
      
          auto make_progress_callback = [](const std::string& tag) {
              return [t = tag](uint64_t transferred, uint64_t transferrable, double estimate) {
                  cout << t << " | PROGRESS: " << transferred << " -> " << transferrable << ' ' << std::fixed
                       << std::setprecision(2) << estimate << endl;
              };
          };
      
          cout << "\n1. upload initial data through upload realm..." << endl;
          SyncTestFile upload_config(harness.session().app()->current_user(), harness.schema(), SyncConfig::FLXSyncEnabled{});
          auto upload_realm = Realm::get_shared_realm(upload_config);
          subscribe_to_all_and_bootstrap(*upload_realm);
          add_objects(upload_realm, "TopLevel", num_objects);
          upload_realm->sync_session()->register_progress_notifier(make_progress_callback(">>> upload"),
                                                                   SyncSession::ProgressDirection::upload, true);
          wait_for_upload(*upload_realm);
      
          SyncTestFile config(harness.app()->current_user(), harness.schema(), SyncConfig::FLXSyncEnabled{});
          auto realm = Realm::get_shared_realm(config);
      
          cout << "\n1.1. add subscription q_1 to download realm and sync..." << endl;
      
          auto table = realm->read_group().get_table("class_TopLevel");
          auto int_col = table->get_column_key("queryable_int_field");
          int split = num_objects / 3;
          auto q_1 = Query(table).between(int_col, 0, split - 1);
      
          realm->sync_session()->register_progress_notifier(make_progress_callback("\t\t<<< download streaming cb"),
                                                            SyncSession::ProgressDirection::download, true);
          realm->sync_session()->register_progress_notifier(make_progress_callback("\t\t<<< download non-stream 1"),
                                                            SyncSession::ProgressDirection::download, false);
      
          {
              auto subs = realm->get_latest_subscription_set().make_mutable_copy();
              subs.insert_or_assign(q_1);
              subs.commit();
          }
          {
              realm->sync_session()->resume();
              realm->get_latest_subscription_set()
                  .get_state_change_notification(sync::SubscriptionSet::State::Complete)
                  .get();
              realm->refresh();
          }
          cout << "... initial sync is completed." << endl;
      
          CHECK(q_1.count() == 3);
      
          cout << "\n2. upload more changes through upload realm..." << endl;
          realm->sync_session()->pause();
          add_objects(upload_realm, "TopLevel", num_objects);
          wait_for_upload(*upload_realm);
      
          cout << "\n2.1. download new changes..." << endl;
          realm->sync_session()->register_progress_notifier(make_progress_callback("\t\t<<< download non-stream 2"),
                                                            SyncSession::ProgressDirection::download, false);
      
          realm->sync_session()->resume();
          wait_for_download(*realm);
          realm->sync_session()->pause();
          realm->refresh();
      
          CHECK(q_1.count() == 6);
      }
      

      </details>

      <details><summary>log output</summary>

      1. upload initial data through upload realm...
      >>> upload | PROGRESS: 6292132 -> 8389486 0.75
      >>> upload | PROGRESS: 7340809 -> 9438163 0.78
      >>> upload | PROGRESS: 8389486 -> 9438163 0.89
      >>> upload | PROGRESS: 9438163 -> 9438163 1.00
      
      1.1. add subscription q_1 to download realm and sync...
                      <<< download non-stream 1 | PROGRESS: 1048759 -> 1048759 1.00
                      <<< download streaming cb | PROGRESS: 1048759 -> 1048759 1.00
                      <<< download streaming cb | PROGRESS: 2097451 -> 2097451 1.00
                      <<< download streaming cb | PROGRESS: 3146143 -> 3146143 1.00
      ... initial sync is completed.
      
      2. upload more changes through upload realm...
      >>> upload | PROGRESS: 9438163 -> 10486840 0.00
      >>> upload | PROGRESS: 10486840 -> 11535517 0.50
      >>> upload | PROGRESS: 11535517 -> 12584194 0.67
      >>> upload | PROGRESS: 12584194 -> 13632871 0.75
      >>> upload | PROGRESS: 12584194 -> 14681548 0.60
      >>> upload | PROGRESS: 13632871 -> 15730225 0.67
      >>> upload | PROGRESS: 14681548 -> 16778902 0.71
      >>> upload | PROGRESS: 15730225 -> 17827579 0.75
      >>> upload | PROGRESS: 16778902 -> 18876256 0.78
      >>> upload | PROGRESS: 17827579 -> 18876256 0.89
      >>> upload | PROGRESS: 18876256 -> 18876256 1.00
      
      2.1. download new changes...
                      <<< download non-stream 2 | PROGRESS: 3146143 -> 3146143 1.00
                      <<< download streaming cb | PROGRESS: 4194820 -> 4194820 1.00
                      <<< download streaming cb | PROGRESS: 5243497 -> 5243497 1.00
                      <<< download streaming cb | PROGRESS: 6292174 -> 6292174 1.00
      

      </details>

      Core version

      Core version: 14.10.3

      #7780 and #7796 changed how the progress is processed, seemingly assuming that the progress from download message is stored in history, but for bootstrap messages it's first stored in bootstrap store, until all messages are received, and only then the changesets are applied. This effectively means that check_progress fetches only last pre-bootstrap value for progress.
      Also, now estimate is treated as bytes unconditionally even for flx and initial history scan on reconnect, effectively mixing the values.

            Assignee:
            thomas.goyne@mongodb.com Thomas Goyne
            Reporter:
            unitosyncbot Unito Sync Bot
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: