Populating a ListView quickly?

Just starting out? Need help? Post your questions and find answers here.
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by PB.

I have 600 lines of data to add to a ListView gadget, totalling 25k.
Adding these 600 lines to the ListView is very slow, making it totally
inaccessible during the populating. If I stick a single WindowEvent()
in there, it makes the gadget visible during the populating, but the
vertical scroll bar doesn't appear until the end of the list, and my
app is "paused" until the end, too. Any ideas on how to do it faster?
API calls? Searching these forums for "listview slow" came up empty.
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by Kale.

maybe using a procedure in a new thread to do the populating, would this help avoid the pause?

--Kale

Getting used to PureBasic and falling in Love! :)
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by Pupil.

I use to fill listview gadget inside a thread, works fine for me, the only thing is that you see when the gadget gets filled(might be that it fills a bit slower because the OS updates the gadget for every item you add), but that's not a big problem, not to me anyway..
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by horst.
Originally posted by PB

I have 600 lines of data to add to a ListView gadget, totalling 25k.
Adding these 600 lines to the ListView is very slow, ...
I just tested this with 25Kb, 600 lines and (buffered) line reading.
It takes 170 ms (@500 Mhz). Is that too slow, or do you have some
other problem?


Horst
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by PB.

> I just tested this with 25Kb, 600 lines and (buffered) line reading.
> It takes 170 ms (@500 Mhz). Is that too slow, or do you have some
> other problem?

In comparison, my 600 lines take 5288 ms to complete (@667 Mhz), so
your measly 170 ms is definitely NOT too slow. :cry:

Kale/Pupil: Doing it in a thread makes no difference, either.

I neglected to mention what I'm doing in my original post: I am actually
calculating strings prior to adding to the ListView -- I'm not filling
it with pre-made strings or with lines from a file. So perhaps all the
string operations are slowing it down. I'll try creating an array
first, then filling the ListView with the array... back in a sec...

Nope, still takes 4777 ms to fill (on average). Guess I'll just have
to live with it.
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by Danilo.

Here is a little Test, so you see its little bit
faster if you lock the gagdet redrawing while
adding the new items.

Code: Select all

DisableDebugger

Procedure Test1()
  ClearGadgetItemList(1)
  While WindowEvent():Wend
  StartTime = TimeGetTime_()
  For a = 1 To 10000
    AddGadgetItem(1,-1,"Entry "+Str(a))
    While WindowEvent():Wend
  Next a
  EndTime   = TimeGetTime_() - StartTime
  AddGadgetItem(2,-1,"Result Test 1: "+StrU(EndTime,2)+" ms")
EndProcedure

Procedure Test2()
  ClearGadgetItemList(1)
  While WindowEvent():Wend
  StartTime = TimeGetTime_()
  For a = 1 To 10000
    AddGadgetItem(1,-1,"Entry "+Str(a))
  Next a
  EndTime   = TimeGetTime_() - StartTime
  AddGadgetItem(2,-1,"Result Test 2: "+StrU(EndTime,2)+" ms")
EndProcedure

Procedure Test3()
  ClearGadgetItemList(1)
  While WindowEvent():Wend
  StartTime = TimeGetTime_()
  SendMessage_(GadgetID(1),#WM_SETREDRAW, #FALSE, 0)
  For a = 1 To 10000
    AddGadgetItem(1,-1,"Entry "+Str(a))
  Next a
  SendMessage_(GadgetID(1),#WM_SETREDRAW, #TRUE, 0)
  EndTime   = TimeGetTime_() - StartTime
  AddGadgetItem(2,-1,"Result Test 3: "+StrU(EndTime,2)+" ms")
EndProcedure

OpenWindow(1,0,0,300,300,#PB_Window_ScreenCentered|#PB_Window_SystemMenu,"")
   CreateGadgetList(WindowID())
   ListViewGadget(1,10,10,280,200)
   ListViewGadget(2,10,225,280,65)

While WindowEvent():Wend
Test1()
While WindowEvent():Wend
Test2()
While WindowEvent():Wend
Test3()

AddGadgetItem(2,-1,"DONE!")
   
Repeat

Until WaitWindowEvent() = #PB_Event_CloseWindow
Procedure Test3() which locks the update
while adding strings is 5 times faster
than Test2() here, so maybe _that_ helps you.

cya,
...Danilo
(registered PureBasic user)
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by geoff.

Pentium III @ 650 MHz 256 MB Win98

Run from edit/debug screen
test 1: 512 ms
test 1: 473 ms
test 3: 94 ms

Run from created executable
test 1: 541 ms
test 2: 476 ms
test 3: 73 ms
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by PB.

> SendMessage_(GadgetID(1),#WM_SETREDRAW, #FALSE / #TRUE, 0)

Nice little tip there, Danilo -- definitely one to remember! :)

However, I have since discovered that it is indeed my string handling
that is causing the delay, because if I pre-render the strings before
enter the AddGadgetItem loop, then all is fine -- the population of
the ListView is virtually instant, with all 600 lines. (When I last
tried this, I was rendering inside the AddGadgetItem loop, which was
silly as it still slowed the loop down). :)

So I need to optimise my string calculations first, which may be a
bit harder to do. Basically, I'm doing a hex display of data in a
ListView gadget. So I have to get each line, render it into a bunch
of hex digits, followed by the ASCII look, and AddGadgetItem it.
You know how any normal hex editor looks? That's what I'm doing,
but it's proving very slow. :cry:
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by geoff.

PB, this sounds exactly like the display of my file edit program written in a different language that I was about to convert to Purebasic.

In this program, I look at the position of the window scroll bar, then dive into the file at this position and draw the screen with the hex digits at that point in the file. Of course, this works OK for files of any size, even 100MB say.

Obviously you couldn't load the whole file into a list view gadget. That's why I was enquiring about the possibility of a scoll bar gadget that could be added to any window, but so far Fred has been rather quite on this one.
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by fred.

ScrollBar are planned, don't worry.

PB: Better write your own buffer->hex number converted, it's easy and will be very fast. Use a pointer to a bye (or long) to do it.


Fred - AlphaSND
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by horst.
Originally posted by PB
[br.. I'm doing a hex display of data ..
Do it without string operations, like this:

Code: Select all

hex.s = "0123456789ABCDEF"

source.s = "Testing ABC" 
product.s = Space(Len(source)*3)

For i = 0 To Len(source)-1
  byte = PeekB(@source+i)
  digit = byte >> 4 : high = PeekB(@hex+digit)
  digit = byte & $F : low  = PeekB(@hex+digit)
  PokeB(@product+p,high) : p+1
  PokeB(@product+p,low) : p+2
Next

MessageRequester("Test",source+Chr(13)+product,0)

End 
Horst
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by PB.

Thanks for the tip, Horst -- I'll work on it to get it displayed how
I like. :)
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by PB.

> Obviously you couldn't load the whole file into a list view gadget.

True, but I'm not using it for files; it's for data under 64k in size.
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by Danilo.

PB, here is a small but fast HEX viewer for you... :)

Code: Select all

;>--------------------------------------------------------
;>--------------------------------------------------------
;
; Pure HexView, by Danilo, 22.01.2003
;
; Thanks go to 'PB' and 'horst' for
; ideas and hints!
;
DisableDebugger ; !!

;
;- Structures
;
Structure Byte
  b.b
EndStructure

;
;- Global variables - arrays - linked lists
;
Dim AsciiChars.b(255)
For a =   0 To 255 : AsciiChars(a) = '.' : Next a
For a =  32 To 127 : AsciiChars(a) = a   : Next a

Dim HexChars.b(16)
PokeS(HexChars(),"0123456789ABCDEF")

;
;- Procedures
;
Procedure AskForEnd()
  If MessageRequester("Exit","Really exit 'Pure HexView' ?",#MB_YESNO|#MB_ICONQUESTION) = #IDYES
     End
  EndIf
EndProcedure


Procedure MOD(a,b)
  ProcedureReturn a-a/b*b
EndProcedure


Procedure FillListViewHex(*mem.Byte,size)
  ;
  ; Fill Listview with hex values
  ; argument 1: memory-pointer for data
  ; argument 2: size of this memory area
  ;
  ;Delay(1)
  StartTime = TimeGetTime_()
  SendMessage_(GadgetID(1),#WM_SETREDRAW, #FALSE, 0)

  A$ = Space(16*4)
  PokeB(@A$+16*3,'|')
  For a = 1 To size/16
    *Text.Byte = @A$
    *RealText.Byte = @A$+16*3+2
    For b = 1 To 16
      *Text\b = HexChars((*mem\b & $FF) >> 4);*offset\b ;PeekB(@HEX$ + offset)
      *Text+1
      *Text\b = HexChars((*mem\b & $F));*offset\b ;PeekB(@HEX$ + offset)
      *Text+2
      *RealText\b = AsciiChars(*mem\b & $FF)
      *RealText+1
      *mem + 1 
    Next b
    AddGadgetItem(1,-1,A$)
  Next a

  b = MOD(size,16)
  If b
    A$ = Space(16*4)
    PokeB(@A$+16*3,'|')
    *Text.Byte = @A$
    *RealText.Byte = @A$+16*3+2
    For a = 1 To b
      *Text\b = HexChars((*mem\b & $FF) >> 4)
      *Text+1
      *Text\b = HexChars((*mem\b & $F))
      *Text+2
      *RealText\b = AsciiChars(*mem\b & $FF)
      *RealText+1
      *mem + 1 
    Next a
    AddGadgetItem(1,-1,A$)
  EndIf
  SendMessage_(GadgetID(1),#WM_SETREDRAW, #TRUE, 0)
  EndTime   = TimeGetTime_() - StartTime
  ;MessageRequester("INFO","Filled ListViewGadget with "+StrU(size,2)+" HEX values in "+StrU(EndTime,2)+" ms",0)
  SetGadgetText(3,StrU(size,2)+" bytes - "+StrU(Size/16+b,2)+" lines generated in "+StrU(EndTime,2)+" ms")
EndProcedure


Procedure LoadFileInListView()
  File$ = OpenFileRequester("Load File","","All Files | *.*",0)
  If File$
    If ReadFile(1,File$)
       size = Lof()
       If size > 200*1024
         If MessageRequester("Warning","Really load BIG file "+GetFilePart(File$)+" ("+StrU(size,2)+" bytes) ??"+Chr(13)+"This can take much time!",#MB_ICONWARNING|#MB_YESNO) = #IDNO 
            CloseFile(1)
            ProcedureReturn
         EndIf
       EndIf
       mem  = AllocateMemory(1,size)
       If mem = 0
         MessageRequester("ERROR","Cant allocate "+StrU(size,2)+" bytes of memory!",#MB_ICONERROR)
         CloseFile(1)
         ProcedureReturn
       EndIf
       ReadData(mem,size)
       CloseFile(1)
       ClearGadgetItemList(1)
       SetGadgetText(3,"File "+GetFilePart(File$)+" loaded ... generating HexView")
       While WindowEvent():Wend
       FillListViewHex(mem,size)
       FreeMemory(1)
    Else
      MessageRequester("ERROR","Cant read file!",#MB_ICONERROR)
    EndIf
  EndIf
EndProcedure

;
;- App START
; 
OpenWindow(1,0,0,760,520,#PB_Window_ScreenCentered|#PB_Window_SystemMenu,"Pure HexView")
   hFont = LoadFont(1,"Lucida Console",14)
   CreateGadgetList(WindowID())
   ListViewGadget(1,0,0,760,500): SendMessage_(GadgetID(1),#WM_SETFONT,hFont,1)
   ButtonGadget(2,640,501,120,20,"Load File in ListView")
   SetGadgetFont(hFont)
   TextGadget(3,0,501,640,20,"",#PB_Text_Center)
   
   While WindowEvent():Wend

   ; produce 32k random data memory for speed testing :)
   mem_data_size = 32*1024
   *mem_data.Byte = AllocateMemory(1,mem_data_size)
   *mem.Byte = *mem_data
   For a = 1 To mem_data_size
     *mem\b = Random($FF)
     *mem + 1
   Next a
   PokeS(*mem_data,"HexViewer for PB")
   For a = 0 To 255 : PokeB(*mem_data+16+a,a) : Next a
   PokeS(*mem_data+mem_data_size-12,"__The End__")

   ; Fill ListViewGadget with random data
   FillListViewHex(*mem_data,mem_data_size)
   
   ; Free random data memory
   FreeMemory(1)

;
;- App message loop
;
Repeat
  Select WaitWindowEvent()
    Case #PB_Event_CloseWindow : AskForEnd()
    Case #PB_Event_Gadget
      Select EventGadgetID()
        Case 2 : LoadFileInListView()
      EndSelect    
  EndSelect
ForEver
;
;- App END
;
;>---------------------------------------------------------
;>---------------------------------------------------------
Speed should be OK because you said you use
max. 64k data size.

cya,
...Danilo
(registered PureBasic user)
BackupUser
PureBasic Guru
PureBasic Guru
Posts: 16777133
Joined: Tue Apr 22, 2003 7:42 pm

Post by BackupUser »

Restored from previous forum. Originally posted by PB.

> PB, here is a small but fast HEX viewer for you... :)

Holy cow -- I wasn't expecting that! Thanks for your time and effort
in coding that... it's really appreciated.
Post Reply