-
Type: Bug
-
Resolution: Works as Designed
-
Priority: Minor - P4
-
None
-
Affects Version/s: 100.1.1
-
Component/s: mongoimport
-
None
-
Environment:mongoimport version: 100.1.1
git version: 8bca136c2a0e0daa3947df31c0624c5615f9aa02
Go version: go1.12.17
os: linux
arch: amd64
compiler: gc
[mediation@talts01mipmed0 log]$ hostnamectl
Static hostname: talts01mipmed0.swi.srse.net
Icon name: computer-vm
Chassis: vm
Virtualization: vmware
Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.6:GA:server
Kernel: Linux 3.10.0-957.10.1.el7.x86_64
Architecture: x86-64mongoimport version: 100.1.1 git version: 8bca136c2a0e0daa3947df31c0624c5615f9aa02 Go version: go1.12.17 os: linux arch: amd64 compiler: gc [ mediation@talts01mipmed0 log]$ hostnamectl Static hostname: talts01mipmed0.swi.srse.net Icon name: computer-vm Chassis: vm Virtualization: vmware Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo) CPE OS Name: cpe:/o:redhat:enterprise_linux:7.6:GA:server Kernel: Linux 3.10.0-957.10.1.el7.x86_64 Architecture: x86-64
I try to import document with `mongoimport` using STDIN
It seems to work only for large amount of documents. Single documents do are not inserted:
````
[mediation@talts01mipmed0 log]$ mongo --norc mip <<< 'db.getCollection("sessions.raw").deleteMany({})'
MongoDB shell version v4.4.1
connecting to: mongodb://127.0.0.1:27017/mip?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("d9219b8e-1d1d-47ab-8955-6881bdb66519") }
MongoDB server version: 4.4.1
bye
[mediation@talts01mipmed0 log]$ mongoimport --db=mip --collection=sessions.raw
2020-11-02T14:06:34.166+0100 connected to: mongodb://localhost/
2020-11-02T14:06:37.166+0100 mip.sessions.raw 0B
2020-11-02T14:06:40.166+0100 mip.sessions.raw 0B
2020-11-02T14:06:43.166+0100 mip.sessions.raw 8B
2020-11-02T14:06:46.166+0100 mip.sessions.raw 8B
2020-11-02T14:06:49.166+0100 mip.sessions.raw 8B
2020-11-02T14:06:52.166+0100 mip.sessions.raw 8B
2020-11-02T14:06:55.166+0100 mip.sessions.raw 8B
^C2020-11-02T14:06:55.665+0100 signal 'interrupt' received; forcefully terminating
[mediation@talts01mipmed0 log]$ mongo --norc mip <<< 'db.getCollection("sessions.raw").countDocuments({})'
MongoDB shell version v4.4.1
connecting to: mongodb://127.0.0.1:27017/mip?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("03ae7a1c-03f1-4070-a0bd-9d8fd7f98b8d") }
MongoDB server version: 4.4.1
0
bye
````
I insert document from syslog-ng, see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/44#TOPIC-1431069
My syslog-ng code is like this:
```
destination d_mongo_snat {
program("/usr/bin/mongoimport --db=mip --collection=sessions.raw"
flush-lines(1)
persist-name("snat")
template("{ \"v\":${VALID}, \"t0\":{\"$$date\":{\"$$numberLong\":\"${BEGINTIME}${.HW.BeginTime}000\"}} }")
);
};
```
The number of documents in my collection is always a multiple of 1000, so it looks like mongoimport imports documents in batches of 1000 but as long as the batch is not full, the data is not written and gets lost.
I was not able to find a way how to flush the buffer. "Ctrl-C" terminates the import without flushing the data.
Any solution?
- split to
-
TOOLS-2773 Add way to manually flush buffer for mongoimport
- Closed