InfoQube IM - Community

Submitted by Tom on 2011/03/31 12:40

CSV text file, tab seperated, no text markers

There are 1594 entries

When I import, I choose correct settings, including deselecting "First row contains field names".

I give each column a field

In step 2 of the import it says, 1 to 635

It imports 635 entries - they are imported correctly but no sign of the rest **- If I change the number to 1594 (actual number of entries) or 1600 there is no difference.

** Two blank items are imported - so it actually only imports 633 entries.

I've had a look at the CSV file in text editor but find no problem (but I'm not an expert).

I've tried this also in the sample file using creating fields for the columns.

Any ideas?

(I could send someone the text file but I wouldnt want to post it here)

Comments

This does look to me like a limit on the number of entries - same problem (same number of entries imported: 633) with a different file (mind you the different file is very similar...)

Anyone else pushed the barriers here?

I did do a search earlier and found something saying there was no limit but cant track it down now....

Also, I just realised that 'CSV' and tab delimited are contradictory terms, but I'll presume you understood

I tried pulling in a 22,000 line csv, and only got it to recognize 5675 lines in step 1 of the import, and then recognize only 883 lines in step 3 of the import dialog.

[quote=KeithB]

I tried pulling in a 22,000 line csv, and only got it to recognize 5675 lines in step 1 of the import, and then recognize only 883 lines in step 3 of the import dialog.[/quote]

OK, so it is limited, or flawed.

odd that I got exactly the same number of lines twice with two (different sized) files

I quickly tested this on three machines that were readily available. Two have 4GB RAM the third has 2GB. An exported database (csv) with 20,000 records stopped at approximately 6000 records on the 4GB machines and 2631 with the 2GB computer. The reason I say approximately with the 4GB machines is because one is running Vista Business and the other 7 Professional, so the memory usage varies. It seems to me that the size that can be processed is limited by the available memory. This is not good.

Making matters worse for me is that I have imported many csv file in the past. Some have been quite large and took a long time to process. Had I not been testing IQ in parallel with my real work (kept in UR, Filemaker, and Ecco), I would have been in trouble. I have groused about this before, but this is too much duplication, too much work, and too many checks that need to be run on IQ to insure data is captured accurately. Aaaaarrrrrrgggggghhhhhhhhhhh!

There. I feel better.

Jon

It's probably a memory management issue while building a huge string. Memory allocation might be failing, even if there is still enough RAM available. Let's let Pierre look at it. That said, while I understand frustration, it's always a good idea to check if what you imported was well imported when you import something. P.S.: When I imported my full evernote DB in IQ, not one item was missing. Same think for all my Outlook tasks and contacts. I can't say the same of the last expensive MS Word version trying to import one of its previous version documents. It just crashed.

I was able to reproduce the issue. Only the first 32000 characters were being imported. Fixed in v0.9.25O

I successfully imported a CSV file containing 22,000 items into the AdrsBook

[quote=Pierre_Admin]

I was able to reproduce the issue. Only the first 32000 characters were being imported. Fixed in v0.9.25O

I successfully imported a CSV file containing 22,000 items into the AdrsBook[/quote]

great

Thank you!

Jon

P.S. Wait a second. Then what would explain the different number of records I was able to import (see the test I performed above)?