Fundamental Limitation of the Tools

The Tools are currently distributed as Golang executables in mongo-tools and MTC.

First of all, it’s generally difficult to “deploy” and test CLI tools in isolation; CLIs intrinsically have to support all platforms across all dependencies.

In addition, it’s not very easy to embed the underlying executables into other products, e.g. Compass and Atlas web UI.

Core Concept - Tools as a Service

The Tools Server is a proposed system, which executes a subset of the Tools’ functionality in the cloud over an HTTP/s service. We could expose all of the Tools in this fashion, but the most salient use cases are in import/export and dump/restore.

The endpoints of the prototype might look something like this:

• /import <args> - import file(s) in JSON/CSV to authenticated DB/collection
• /export <args> - export authenticated DB/collection to JSON/CSV
• /dump <args> - dump BSON file(s) corresponding to DB/collection in specified file sink
• /restore <args> - restore given BSON files to authenticated DB/collection

This design gives rise to the following possible features:

• Provide import/export functionality over web UI and Compass, e.g. “Upload File” and “Save Cluster as {CSV, JSON}” buttons
• This would be useful in analytics workflows, where users rely on GUIs but tend to need this functionality
• Use import to provide live sync between ADL and any MDB instance
• Use dump/restore to provide live backup of any MDB instance to the cloud, i.e. an S3 bucket 

A possible prototype of the system would be a generic implementation of dump/restore and import/export in Realm and/or AWS Lambda. 

Short-Term Benefits

**In the past, we've considered the following Epics:

• TOOLS-2072 - rationalize CLI options and formalize interface
• TOOLS-2279 - UI quick wins

These give rise to higher-level problems in the Tools and the ecosystem, i.e. “What would a stable and succinct interface over the Tools look like? How do we implement that?”

This problem in particular is solved much more easily by versioning and distributing a HTTP/s-based Tools API and leaving the functionality embeddable to various interfaces. We could trivially support the CLI, i.e. by using the HTTP/s endpoints, for compatibility, if we wanted.

Fundamentally, decoupling the Tools’ functionality as an API would result in better testing and reproducibility, i.e. using various Cloud services.

Long-Term Benefits

If we support a stable implementation of the Tools as a service, we could enjoy the following:

• More features are possible to the users under this architecture
• GUI button on Compass/web UI to “Download File as CSV” (and so on) using export
• Cloud service to back up an Atlas cluster to ADL using restore
• Teams and orgs that want to use the Tools in the future have an easier time doing so, since HTTP/s is embeddable into many services
• Cloud services are anecdotally more easily authenticated, tested and versioned than CLIs, thanks to MDB itself as a platform