WT-7514 introduces a `__wt_btree_switch_object` function, that tells the block manager that the current id has changed.  This might be too large a granularity, and that new function has a flags field that can be used to fix this.  In particular we should consider breaking this one call down into several calls that notify the block manager to do certain things:

• READONLY: disallow writes to the current object id (we're about to switch, and we don't want any more eviction writes to this object).  This would occur after a checkpoint for a file has completed.  We don't want writes because the file is now immutable, and we want to ship it some network based storage.  We'd need some sort of marker/lock in the block manager to prevent writes.
• WRITEABLE: mark the given object id as the current writeable object id.  This is basically what we're doing now, it's called when the current id has changed.  In the Block manager we could open the new file for writing.  It's also possible to open it lazily, not sure if that's a good idea.  If we're holding a lock after disallowing writes, we'd release that lock.
• CLOSE_DISALLOW: close the given object id, and don't allow reopens yet.  This would be called after the object has been copied, and when it is about to be moved out of the WT home directory.
• OPEN_ALLOW: allow opens on the given object id.  This is called after the object is moved out of the WT home directory and it should be opened from cloud storage or its cache.

Not clear that we need the separation for the first two points: In favor is that it would be easier for the upper layer to track whether this is the initial setting of the writeable object id (in which case the second call only is made), or whether it's a switch (both calls are made).  Arguing against is that maybe we'll want to do this under a single lock in the block manager, it makes things easier, rather than holding a lock over two calls.

The last two points (CLOSE_DISALLOW, OPEN_ALLOW) resolve an issue about moving a file.  On Windows at least, we can't move or remove a file that is open.  On UNIX systems we can, but it's not always good form.  The cleanest way would be to call CLOSE_DISALLOW, update metadata to remove the file, then remove the file itself from the WT home, and actually move it to the cache for the storage source, finally call OPEN_ALLOW.  Not having this separation means this is the only place in WT where we remove or move a file that we allow to be open.

See FIXME in the code for this.