-
Type:
New Feature
-
Resolution: Gone away
-
Priority:
Unknown
-
None
-
Affects Version/s: None
-
Component/s: GridFS
-
None
I am writing a repository project. The repo has grown quite large. At some point find chunks query is exceeding the memory limit. I solved this locally by down loading the driver source code and add the allowDiskUse to the cursor creation in GridFSDownloadStreamImpl
class GridFSDownloadStreamImpl extends GridFSDownloadStream { private final ClientSession clientSession; private final GridFSFile fileInfo; private final MongoCollection<Document> chunksCollection; private final BsonValue fileId; private final long length; private final int chunkSizeInBytes; private final int numberOfChunks; private final boolean allowDiskUse; private MongoCursor<Document> cursor; private int batchSize; private int chunkIndex; private int bufferOffset; private long currentPosition; private byte[] buffer = null; private long markPosition; private final ReentrantLock closeLock = new ReentrantLock(); private final ReentrantLock cursorLock = new ReentrantLock(); private boolean closed = false; GridFSDownloadStreamImpl(@Nullable final ClientSession clientSession, final GridFSFile fileInfo, final MongoCollection<Document> chunksCollection) { this(clientSession, fileInfo, chunksCollection, false); } GridFSDownloadStreamImpl(@Nullable final ClientSession clientSession, final GridFSFile fileInfo, final MongoCollection<Document> chunksCollection, final Boolean allowDiskUse) { this.clientSession = clientSession; this.fileInfo = notNull("file information", fileInfo); this.chunksCollection = notNull("chunks collection", chunksCollection); this.allowDiskUse = allowDiskUse;
private MongoCursor<Document> getCursor(final int startChunkIndex) { FindIterable<Document> findIterable; Document filter = new Document("files_id", fileId).append("n", new Document("$gte", startChunkIndex)); if (clientSession != null) { findIterable = chunksCollection.find(clientSession, filter).allowDiskUse(isAllowDiskUse()); } else { findIterable = chunksCollection.find(filter).allowDiskUse(isAllowDiskUse()); } return findIterable.batchSize(batchSize).sort(new Document("n", 1)).iterator(); }
Then I added the choice as an optional argument to all the download functions. Existing signatures default to false so behavior doesn't change
example:
/**
* Opens a Stream from which the application can read the contents of the stored file specified by {@code id}.
*
* @param id the ObjectId of the file to be put into a stream.
* @return the stream
*/
GridFSDownloadStream openDownloadStream(ObjectId id);
/**
* Opens a Stream from which the application can read the contents of the stored file specified by {@code id}.
*
* @param id the ObjectId of the file to be put into a stream.
* @param allowDiskUse boolean allow disk use in chunk query
* @return the stream
*/
GridFSDownloadStream openDownloadStream(ObjectId id, boolean allowDiskUse);