Submitted by jimspoon on 2020/12/23 13:10
I successfully imported some 140,000 items from a CSV file (saved from Excel) to a test IQ database.  The items had only two fields - Item and EntryDate.
 
Now that I have them in my "test" IQ database, and things are working well, I thought I would try copying/exporting the items from the test database to my "main" IQ file.
 
The problem is, the copy/export functions all pertain to Selected items.  So you have to display them in a grid and then select them before you can attempt the copy/export.  
 
I can't load all the 140,000 items I want to copy/export in a grid.  I selected "load all items" maybe an hour ago but the grid still hasn't loaded, and the hourglass is still showing.
 
I could do this export in several pieces, by loading fewer items into the grid, doing an export, then dong another batch, etc. but that's obviously far from ideal.
 
Actually, I can do the import from the original CSV file, but it would be good if it would work from within IQ too. 
 
The grid load mechanism seems to be the bottleneck.  It seems the copy/export could work if the grid load could be bypassed - e.g. if an export could be performed from the "source bar" criteria, the specification of the columns in the grid, without trying to all the matching items in a grid.
 
UPDATE: Looking at https://infoqubeim.com/drupal5/?q=node/321 , I was wondering if I could use the Export to Excel function to export items to Excel without actually loading them all in a grid first.  The page says that items "belonging to the grid will be exported", i .e. "those that meet the grid source/filter".  I tried to do an export to excel after loading only the first 1000 items of the grid, hoping that ALL items matching the source/filter would be exported, not just the first 1000.  But immediately (i.e. with no delay) I got this error:
 
 
This "journal import.xls" file had a size over 2,000,000 bytes but there was no visible data inside.
 

Comments

Hi Jim,
 
Indeed, IQ needs improvements to better handle large lists. Did you try to create an ODC file and open it in Excel ?
 
That said, I think the import method which should be the fastest is Import > Delimited File, so perhaps the original .csv file is still the best method...
 
 
HTH !
 
Pierre_Admin
IQ Designer
 

jimspoon

2020/12/23 21:50

In reply to by Pierre_Admin

Wow !!  Excel fetched all 140,000 rows instantly.  I couldn't believe it.  I will study up on how to use ODC files.  Thanks Pierre very much for the tip.
 
There is a related problem that I see with having such a large number of items.  Say you want to to modify a field value on a very large number of items, more than IQ can load in the grid at one time.  Earlier today I was trying to figure out how I might do that via a row equation to specify both the items to be operated on (using IF statements?) and the value to be inserted into the field for those items.  Maybe ODC can help there too.
 
 

Pierre_Admin

2020/12/23 22:32

In reply to by jimspoon

Hi Jim,
[quote]
Maybe ODC can help there too.[/quote]
 
I don't think so, as ODC can only read data, not write...
 
but as you say, you can set up a field equation and apply it to all items in the database (using the small equation refresh button)
 
Pierre_Admin
IQ Designer