InfoQube IM - Community

Submitted by jimspoon on 2018/03/19 11:09

Has anybody else tried to import a very large number of items into IQ from a CSV file? My attempts have never succeeded.

I think IQ would make a great bookmark manager, and I would also like to have my boomarks integrated into my other IQ data. So I exported my data from Linkman to a TSV file. (Linkman a great bookmark manager, but I've had lots of trouble getting exports to work properly.) Used Excel to convert the TSV into a CSV file. Had to do some trickery in Excel to get a CSV file with only three columns, and eliminate a bunch of empty columns.

So then I have a CSV with only three columns for each bookmark - title, url, and date. Created a new IQ file, and created three new fields - BookmarkTitle (text), BookmarkURL (text, hyperlink), and BookmarkDate (date). Then I tried to import the CSV file as non-hierarchical data. I reached the point shown in the screenshot below, which seems to indicate that IQ had received the CSV data correctly:

Then I clicked then Next button. Then I get to the following window. All the records to be imported are empty - the CSV data did not make it to this stage:

If I then click the import, the specified number of records are indeed created in my IQ database, but they contain no data from the CSV file.

Anyone have an idea what's going wrong? I've tried changing up the data, the import conditions in various ways, but I always end up at the "empty rows" window.

Comments

Hi Jim,

Did you try to use Tab-delimited format? Perhaps you'll have more luck.

I'll try with a test file and report back

HTH !

Pierre_Admin

IQ Designer

Hi !

I found the issue. This was really old code and an integer counter was used, as you had more than 32K items, it failed. Fixed in the next version

Pierre_Admin

IQ Designer

cool! I'll experiment.

63195 bookmarks... Jim, isn't that excessive ???

Also, import is faster if you uncheck "Current Grid" and manually set a Y/N field for your bookmarks. Finally, I'd use the Item field for the bookmark text, and the built-in URL field for the URL

Pierre_Admin

IQ Designer

haha .. yes, but I'm an excessive kind of guy. Infoholic. I imagine that quite a few of us IQ buffs are into stretching the limits!

I agree your field assignment suggestions are better than one I picked. Though Item doesn't have to be one of the fields for an item, it does make good sense for lots of reasons. I'll try out adding a Y/N "Bookmark" field.

still experimenting ... this time I tried importing 28K items from the CSV file. Indeed 28K items were created (operation took a very long time to complete), but only only 10K - 12K of the values were imported for each of the three import fields. Just want to pass this along.

Might be a fault in the CSV data - here are three sample lines from the CSV file.

Carnival of Souls (1962) - Overview - TCM.com,http://www.tcm.com/tcmdb/title/70341/Carnival-of-Souls/,10/30/2016 00:30

David Copperfield (1935) - Overview - TCM.com,http://www.tcm.com/tcmdb/title/72384/David-Copperfield/,04/07/2017 20:34

The Phantom Carriage (1921) - Overview - TCM.com,http://www.tcm.com/tcmdb/title/724173/Phantom-Carriage-The/,10/23/2016 23:52

Maybe each of the fields needs to be delimited with double-quotes?

Hi Jim,

I've improved the import process speed. A couple of minutes is now sufficient to import 60K items

Perhaps try Tab delimited instead of comma (as a comma in the bookmark name will throw off the importer.)

Another option is to put the bookmark title as the last field

Pierre_Admin

IQ Designer

really looking forward to trying this out in the next update!