GetURL source code

Share your advanced PureBasic knowledge/code with the community.
DEU.exe
User
User
Posts: 13
Joined: Sun Oct 19, 2003 11:21 am
Location: www

Post by DEU.exe »

First, thanks a lot for this great lib! It works perfectly.

But...

Could someone be so kind to explain how to use it to download a file to memory?

urlfile\savepath=*mem or @string

Is it possible?
pIII@1Ghz, wXP, 512MB, pb3.93full
GPI
PureBasic Expert
PureBasic Expert
Posts: 1394
Joined: Fri Apr 25, 2003 6:41 pm

Post by GPI »

ok, i change my code (note: This code is *NOT* the source of the getURL-Lib. At the moment i must say: DON'T USE THE LIB IT!)

Follow this do my code better than the GetURL-Source:

* abort Download is possible

* Can handle error-messages
GetURL don't return anything, when a 404 (file not found) happend. And getURL-detect can detect a 404, where no one is. And when a error happend, that geturl don't know, the startet thred of the lib run eternaly.

* Download more than one file at the same time
(ok, it is possible with my code, but it can be a little bit complicate to do this...)

* Read and interpretate the date of a Internet file

* HTTP-Get command header is analysed
GetURL need a HTTP-Head-Command first and then he cut the Get-Header (and in bad cases, the header aren't equal; theoretical). My code only send only a get-command and analyse the header.

* support "splitted" header
When the buffer length is small and the header is big, geturl-lib can't handle this. My code can.

* Proxy without password and Web-User/Proxy-User Name+Password supported.
The Source above of the GetURL-Lib can't do this (with PB3.92), because of a bad call of Base64Encoder().

I have tested proxy (without password) and side with authorization. But i can't test it with a proxy with password.
It would be nice, when somebody can test this.

Code: Select all

;-Create and Read file with API
Structure API_FileHandle
  FHandle.l
  Buffer.l
  BufferLen.l
  ReadPos.l
  DataInBuffer.l
  readed.l
EndStructure

Procedure API_CloseFile(*File.API_FileHandle)
  If *File\FHandle<>#INVALID_HANDLE_VALUE
    CloseHandle_(*File\FHandle)
  EndIf
  If *File\Buffer
    FreeMemory(*File\Buffer):*File\Buffer=0
  EndIf
EndProcedure
Procedure API_FileCreate(*File.API_FileHandle,File$)
  *File\FHandle=CreateFile_(File$,#GENERIC_WRITE	,0,0,#CREATE_ALWAYS	,#FILE_ATTRIBUTE_NORMAL,0)
  If *File\FHandle=#INVALID_HANDLE_VALUE
    ProcedureReturn #False
  Else
    ProcedureReturn #True
  EndIf
EndProcedure
Procedure API_WriteData(*File.API_FileHandle,*Buffer,len)
  If WriteFile_(*File\FHandle,*Buffer,len,@written,0)
    ProcedureReturn written
  Else
    ProcedureReturn 0
  EndIf
EndProcedure

;-Help

Procedure.s MyStringField(a$,Index,Seperator$)
  AkPos=0
  LenSeperator=Len(Seperator$)
  LenA=Len(a$)
  For i=1 To Index
    a=FindString(a$,Seperator$,AkPos+LenSeperator)
    If a>0
      If AkPos>0
        StartPos=AkPos+LenSeperator
      Else
        StartPos=1
      EndIf
      AkPos=a
    ElseIf AkPos<LenA
      If AkPos>0
        StartPos=AkPos+LenSeperator
      Else
        StartPos=1
      EndIf
      AkPos=LenA+1
    Else
      i=Index
      AkPos=0
      StartPos=0
    EndIf
  Next
  
  If StartPos
    ProcedureReturn Mid(a$,StartPos,AkPos-StartPos)
  Else
    ProcedureReturn ""
  EndIf
EndProcedure

;-Structures and Constants

Structure HTTPGetId
  url$
  server$
  File$
  FileName$
  Port.l
  wenc$
  ProxyServer$
  ProxyPort.l
  penc$
  cookie$
  location$
  ConnectionID.l
  ErrorMessage$
  FileSize.l
  Date.l
  URLError.l
  Buffer.l
  BufferLength.l
  Realm$ 
 
  InHeader.l
  FirstLine.l
  currentline$
  lastchar.l
  eol.l
  Received.l
  Typ$
  
  FileHandle.API_FileHandle
EndStructure
#URLError_OK=200
#URLError_CantCreateFile=1
#URLERROR_OutOfMemory=2


;-URL-Commands
;{Day,Month
Dim wkday$(6)
Dim weekday$(6)
Dim Month$(12)
wkday$(1)="MON" 
wkday$(2)="TUE" 
wkday$(3)="WED"
wkday$(4)="THU" 
wkday$(5)="FRI"
wkday$(6)="SAT"
wkday$(0)="SUN"
weekday$(1)="MONDAY"
weekday$(2)="TUESDAY"
weekday$(3)="WEDNESDAY"
weekday$(4)="THURSDAY"
weekday$(5)="FRIDAY"
weekday$(6)="SATURDAY"
weekday$(0)="SUNDAY"
Month$(1)="JAN"
Month$(2)="FEB"
Month$(3)="MAR"
Month$(4)="APR"
Month$(5)="MAY"
Month$(6)="JUN"
Month$(7)="JUL"
Month$(8)="AUG"
Month$(9)="SEP"
Month$(10)="OCT"
Month$(11)="NOV"
Month$(12)="DEC"
;}
Procedure.s HTTP_CreateDate(Date)
  ProcedureReturn FormatDate(wkday$(DayOfWeek(Date))+" %dd "+Month$(Month(Date))+" %yyyy %hh:%ii:%ss GMT",Date)
EndProcedure
Procedure HTTP_AnalyseDate(TestTime$)
  TestTime$=ReplaceString(Trim(UCase(TestTime$)),"  "," ")
  Day=0:Month$="":Year=0:Time$=""
  For i=0 To 6
    If Left(TestTime$,4)=wkday$(i)+","                        ;{"rfc1123-Date"
      Day=Val(StringField(TestTime$,2," "))
      Month$=StringField(TestTime$,3," ")
      Year=Val(StringField(TestTime$,4," "))
      Time$=StringField(TestTime$,5," ")
      Break
      ;}
    ElseIf Left(TestTime$,Len(weekday$(i))+1)=weekday$(i)+"," ;{"rfc850-Date"
      SubTime$=StringField(TestTime$,2," ")
      Day=Val(StringField(SubTime$,1,"-"))
      Month$=StringField(SubTime$,2,"-")
      Year=Val(StringField(SubTime$,3,"-"))
      If Year>80:Year+1900:Else:Year+2000:EndIf
      Time$=StringField(TestTime$,3," ")
      Break
      ;}
    ElseIf Left(TestTime$,4)=wkday$(i)+" "                    ;{"asctime-Date"
      Day=Val(StringField(TestTime$,3," "))
      Month$=StringField(TestTime$,2," ")
      Year=Val(StringField(TestTime$,5," "))
      Time$=StringField(TestTime$,4," ") 
      Break
      ;}
    EndIf
  Next
  For i=1 To 12
    If Month$(i)=Month$ : Month=i:Break : EndIf
  Next
  Date=ParseDate("%hh:%ii:%ss",Time$)
  Hour=Hour(Date)
  Min=Minute(Date)
  Sec=Second(Date)
  ProcedureReturn Date(Year,Month,Day,Hour,Min,Sec)
EndProcedure  
Procedure.s HTTP_CryptedUserPass(user$,pass$)
  If user$
    conc$=user$+":"+pass$ 
    OutputBuffer = AllocateMemory(Len(conc$)*2) 
    Base64Encoder(conc$,Len(conc$),OutputBuffer,Len(conc$)*2) 
    penc$=PeekS(OutputBuffer) 
    FreeMemory(OutputBuffer)
  Else
    penc$=""
  EndIf
  ProcedureReturn penc$
EndProcedure

Declare HTTP_ReConnect(*Handle.HTTPGetId)
Declare HTTP_ChangeURL(*Handle.HTTPGetId,url$)
Declare HTTP_CloseURL(*Handle.HTTPGetId)

;internal
Procedure.s HTTP_CreateRequestString(*Handle.HTTPGetId)
  If *Handle\ProxyServer$
    com$=UCase(*Handle\Typ$)+" "+*Handle\url$+" HTTP/1.0"+Chr(13)+Chr(10)
  Else
    com$=UCase(*Handle\Typ$)+" "+*Handle\File$+" HTTP/1.0"+Chr(13)+Chr(10)
  EndIf
  com$+"Host: "+*Handle\server$+Chr(13)+Chr(10)
  com$+"Accept: */*"+Chr(13)+Chr(10)
  
  com$+"User-Agent: PBGPIHTTPs"+Chr(13)+Chr(10)
  If *Handle\ProxyServer$<>"" And *Handle\penc$
    com$+"Proxy-Authorization: Basic "+*Handle\penc$+Chr(13)+Chr(10)
  EndIf
  If *Handle\wenc$<>""
    com$+"Authorization: Basic "+*Handle\wenc$+Chr(13)+Chr(10)
  EndIf
  If *Handle\cookie$<>""
    com$+"Cookie: "+*Handle\cookie$+Chr(13)+Chr(10)
  EndIf
  If *Handle\location$<>""
    com$+"Location: "+*Handle\location$+Chr(13)+Chr(10)
  EndIf
  com$+Chr(13)+Chr(10)
  Debug com$
  ProcedureReturn com$
EndProcedure
;Get Information
Procedure HTTP_IsHeaderReceived(*Handle.HTTPGetId)
  ProcedureReturn #True-*Handle\InHeader
EndProcedure
Procedure HTTP_GetProgress(*Handle.HTTPGetId); Header must be received!
  If *Handle\FileSize
    ProcedureReturn *Handle\Received*100/*Handle\FileSize
  EndIf
EndProcedure
Procedure HTTP_GetError(*Handle.HTTPGetId); Header must be received!
  ProcedureReturn *Handle\URLError
EndProcedure
Procedure.s HTTP_GetErrorMessage(*Handle.HTTPGetId); Header must be received!
  ProcedureReturn *Handle\ErrorMessage$
EndProcedure 
Procedure.s HTTP_GetNewLocation(*Handle.HTTPGetId); Header must be received!
  ProcedureReturn *Handle\location$
EndProcedure
Procedure.s HTTP_GetAuthenticateRealm(*Handle.HTTPGetId); Header must be received!
  ProcedureReturn *Handle\Realm$
EndProcedure
Procedure HTTP_GetFileSize(*Handle.HTTPGetId); Header must be received!
  ProcedureReturn *Handle\FileSize
EndProcedure
Procedure HTTP_GetFileDate(*Handle.HTTPGetId); Header must be received!
  ProcedureReturn *Handle\Date
EndProcedure
;Change Settings
Procedure HTTP_ChangeURL(*Handle.HTTPGetId,url$)
  ;Port finden
  s_start=FindString(url$,":",7)
  s_end=FindString(url$,"/",s_start)
  If s_start And s_end
    port$=Mid(url$,s_start+1,s_end-(s_start+1))
  EndIf
  ;Servername finden
  For a=1 To Len(url$)
    s_start=FindString(url$,"//",1)
    s_end=FindString(url$,"/",s_start+2)
    server$=Mid(url$,s_start+2,s_end-(s_start+2))
    If port$<>""
      server$=ReplaceString(server$,":"+port$,"",1)
    EndIf
  Next
  ;standard-Name
  If Port=0: Port=80 :EndIf
  ;Dateiname
  File$=ReplaceString(url$,"http://"+server$,"",1)
  
  *Handle\url$=url$
  *Handle\Port=Port
  *Handle\server$=server$
  *Handle\File$=File$
EndProcedure
Procedure HTTP_ChangeWWWAuthenticate(*Handle.HTTPGetId,wenc$)
  *Handle\wenc$=wenc$
EndProcedure
Procedure HTTP_ChangeProxyAuthenticate(*Handle.HTTPGetId,penc$)
  *Handle\penc$=penc$
EndProcedure
;Commands
Procedure HTTP_Connect_(*Handle.HTTPGetId,LocalFile$,url$,wenc$,ProxyServer$,ProxyPort,penc$,BufferLength,Typ$);internal
  If BufferLength<=0: BufferLength=10240 :EndIf
  
  HTTP_ChangeURL(*Handle.HTTPGetId,url$)
  
  If ProxyPort=0: ProxyPort=3196 :EndIf
  
  ;{ Infos speichern
  *Handle\FileName$=LocalFile$
  *Handle\wenc$=wenc$
  *Handle\ProxyServer$=ProxyServer$
  *Handle\ProxyPort=ProxyPort
  *Handle\penc$=penc$
  *Handle\cookie$=""
  *Handle\Buffer=0
  *Handle\BufferLength=BufferLength
  *Handle\Typ$=Typ$
  ;}
  
  ProcedureReturn HTTP_ReConnect(*Handle)
EndProcedure
Procedure HTTP_OpenUrl(*Handle.HTTPGetId,LocalFile$,url$,wenc$,ProxyServer$,ProxyPort,penc$,BufferLength)
  ProcedureReturn HTTP_Connect_(*Handle.HTTPGetId,LocalFile$,url$,wenc$,ProxyServer$,ProxyPort,penc$,BufferLength,"GET")
EndProcedure
Procedure HTTP_UrlInfo(*Handle.HTTPGetId,url$,wenc$,ProxyServer$,ProxyPort,penc$,BufferLength)
  ProcedureReturn HTTP_Connect_(*Handle.HTTPGetId,"",url$,wenc$,ProxyServer$,ProxyPort,penc$,BufferLength,"HEAD")
EndProcedure
Procedure HTTP_ReConnect(*Handle.HTTPGetId); Usefull, when url is moved or authenticate is needed
  HTTP_CloseURL(*Handle)
  
  *Handle\location$=""
  *Handle\FileSize=0
  *Handle\Date=0
  *Handle\ErrorMessage$=""
  *Handle\InHeader=#True
  *Handle\FirstLine=#True
  *Handle\currentline$=""
  *Handle\lastchar=0
  *Handle\eol=0
  *Handle\URLError=0
  *Handle\Realm$=""
  *Handle\Received=0
  Debug *Handle\server$
  Debug *Handle\Port
  If *Handle\ProxyServer$<>""
    *Handle\ConnectionID = OpenNetworkConnection(*Handle\ProxyServer$, *Handle\ProxyPort)
  Else
    *Handle\ConnectionID = OpenNetworkConnection(*Handle\server$, *Handle\Port)
  EndIf
  
  If *Handle\ConnectionID
    com$=HTTP_CreateRequestString(*Handle)
    SendNetworkData(*Handle\ConnectionID,@com$,Len(com$))
    ProcedureReturn #True
  Else
    ProcedureReturn #False
  EndIf
EndProcedure
Procedure HTTP_CloseURL(*Handle.HTTPGetId); little note: don't earse file-info (HTTP_GET*)
  If *Handle\ConnectionID
    CloseNetworkConnection(*Handle\ConnectionID)
    *Handle\ConnectionID=#False
  EndIf
  If *Handle\Buffer
    FreeMemory(*Handle\Buffer)
    *Handle\Buffer=0
  EndIf
  API_CloseFile(*Handle\FileHandle)
EndProcedure
Procedure HTTP_ReceiveData(*Handle.HTTPGetId) ; #True=Not ready  #False=AllDatasRecived
  Result=#True
  NCEvent = NetworkClientEvent(*Handle\ConnectionID)
  If NCEvent=2 ; Raw-data
    If *Handle\Buffer=0
      *Handle\Buffer=AllocateMemory(*Handle\BufferLength)
      If *Handle\Buffer=0
        *Handle\URLError=#URLERROR_OutOfMemory
        ProcedureReturn #False
      EndIf
    EndIf
    
    InBuffer=ReceiveNetworkData(*Handle\ConnectionID,*Handle\Buffer,*Handle\BufferLength)
    *start.BYTE=*Handle\Buffer
    If *Handle\InHeader ;{ Header analysieren
      While InBuffer
        char=(*start\b & $FF):InBuffer-1:*start+1
        Select char
          Case #CR;: #crlf$
          Case #LF
            If *Handle\lastchar=#CR; Ende der Zeile entdeckt!
              If *Handle\eol; zum zweiten mal -> nun daten
                *Handle\InHeader=#False
                ;{-Header eingelesen ->Auswerten
                If *Handle\URLError<>200 
                  Result=#False
                Else;OK, datei öffnen
                  If *Handle\FileSize And *Handle\Typ$="GET" And *Handle\FileName$
                    If API_FileCreate(*Handle\FileHandle,*Handle\FileName$)=#False
                      *Handle\URLError=#URLError_CantCreateFile
                      result=#false
                    EndIf
                  Else
                    Result=#False
                  EndIf
                EndIf
                ;}
                Break
              Else
                *Handle\eol=#True
                Debug *Handle\currentline$
                If *Handle\FirstLine;{-Status-code auslesen
                  *Handle\FirstLine=#False
                  a1=FindString(*Handle\currentline$," ",0)
                  a2=FindString(*Handle\currentline$," ",a1+1):If a2=0:a2=Len(*Handle\currentline$)+1:EndIf
                  ;HTTPVer:Left(*Handle\currentline$,a1-1)
                  *Handle\URLError=Val(Mid(*Handle\currentline$,a1+1,a2-a1-1))
                  *Handle\ErrorMessage$=Right(*Handle\currentline$,Len(*Handle\currentline$)-a2)
                  ;}
                Else;{-Header-Zeile analysieren
                  a=FindString(*Handle\currentline$,":",1)
                  Variable$=Trim(Left(*Handle\currentline$,a-1))
                  Set$=Trim(Right(*Handle\currentline$,Len(*Handle\currentline$)-a))
                  Select UCase(Variable$)
                    Case "PROXY-AUTHENTICATE";{  Realm auslesen
                      a=FindString(Set$,"realm=",0)
                      b=FindString(Set$,#DQUOTE$,a+7)
                      If a And b
                        *Handle\Realm$=Mid(Set$,a+7,b-a-7)
                      EndIf
                      ;}
                    Case "WWW-AUTHENTICATE";{    Realm auslesen
                      a=FindString(Set$,"realm=",0)
                      b=FindString(Set$,#DQUOTE$,a+7)
                      If a And b
                        *Handle\Realm$=Mid(Set$,a+7,b-a-7)
                      EndIf
                      ;}
                    Case "LOCATION":            *Handle\location$=Set$
                    Case "SET-COOKIE";{          Cookie suchen
                      a=FindString(Set$,";",0):If a=0: a=Len(Set$)+1 :EndIf
                      *Handle\cookie$=Left(Set$,a-1)
                      ;}
                    Case "CONTENT-LENGTH":      *Handle\FileSize=Val(Set$)
                    Case "LAST-MODIFIED":       *Handle\Date=HTTP_AnalyseDate(Set$)
                  EndSelect    
                  ;}
                EndIf
                *Handle\currentline$=""
              EndIf
            EndIf
          Default
            *Handle\eol=#False
            *Handle\currentline$+Chr(char)
        EndSelect
        *Handle\lastchar=char
      Wend
      ;}
    EndIf
    If *Handle\InHeader=#False ;{
      *Handle\Received+InBuffer
      API_WriteData(*Handle\FileHandle,*start,InBuffer)
      If *Handle\Received=*Handle\FileSize
        Result=#False;fertig
        API_CloseFile(*Handle\FileHandle)
      EndIf
      ;}
    EndIf
  EndIf
  
  ProcedureReturn Result
EndProcedure


;- example

;url.s="http://192.168.1.1/index.html"
url.s="http://www.sedtech.com/isedquickpdf/downloads/4.35/iSQP0435DLL.zip"
;url.s="http://gpihome.de/Crillion/English_asdfdafs_.txt"

;File$="C:\test.txt"
File$="c:\test.zip"

; ;Proxy required?
; ProxyServer$="192.168.1.1"
; ProxyPort=8080
; ProxyServer$="195.140.251.142"
; ProxyPort=8080
;ProxyPass$=CreateCryptedPass("user","pass")

; ;user and password required?
; WebPass$=CreateCryptedPass(user$,pass$)



If InitNetwork()
  
  OpenConsole()
  PrintN("***Getting file information***")
  If HTTP_UrlInfo(Info.HTTPGetId,url,WebPass$,ProxyServer$,ProxyPort,ProxyPass$,0)
    PrintN("connect...")
    While HTTP_ReceiveData(Info); wait for header
      Delay(100)
    Wend
    HTTP_CloseURL(Info)
    Select HTTP_GetError(Info)
      Case #URLError_OK
        PrintN("FileSize:"+Str(HTTP_GetFileSize(Info)))
        PrintN("FileDate:"+FormatDate("%dd.%mm.%yyyy %hh:%ii:%ss",HTTP_GetFileDate(Info)))
      Case #URLERROR_OutOfMemory:PrintN("Out of memory")
      Case #URLError_CantCreateFile:PrintN("Can't create file")
        ;for all other errors: See download
      Default
        PrintN(Str(HTTP_GetError(Info))+" "+HTTP_GetErrorMessage(Info))
    EndSelect
  EndIf
  
  
  PrintN("")
  PrintN("***Getting file***")
  If HTTP_OpenUrl(Get.HTTPGetId,File$,url,WebPass$,ProxyServer$,ProxyPort,ProxyPass$,0)
    Repeat
      PrintN("Connect...")
      ReTry=#False:Progress=0
      While HTTP_ReceiveData(Get)
        a=HTTP_GetProgress(Get)
        If a<>Progress
          PrintN("Progress: "+Str(a)+"%")
          Progress=a
        EndIf
        ;if want abort
        ;  http_closeurl(get$)
        ;  printn("Abort")
        ;  break 2
        ;endif
        Delay(100)
      Wend
      HTTP_CloseURL(Get)
      
      Select HTTP_GetError(Get)
        Case #URLError_OK:PrintN("Download complete")
        Case #URLERROR_OutOfMemory:PrintN("Out of memory")
        Case #URLError_CantCreateFile:PrintN("Can't create file")
        Case 301;Moved permanently
          PrintN("Moved permanently:"+HTTP_GetNewLocation(Get))
          HTTP_ChangeURL(Get,HTTP_GetNewLocation(Get))
          If HTTP_ReConnect(Get); neu anfordern
            ReTry=#True
          Else
            PrintN("Can't connect")
          EndIf
        Case 302;Moved temporarily
          PrintN("Moved temporarily:"+HTTP_GetNewLocation(Get))
          HTTP_ChangeURL(Get,HTTP_GetNewLocation(Get))
          If HTTP_ReConnect(Get); neu anfordern
            ReTry=#True
          Else
            PrintN("Can't connect")
          EndIf
        Case 305;Use Proxy
          PrintN("Use Proxy:"+HTTP_GetNewLocation(Get))
        Case 401;unauthorized
          PrintN("Unauthorized:"+HTTP_GetAuthenticateRealm(Get))
          Print("  UserName:"):user$=Input():PrintN("")
          Print("  Password:"):pass$=Input():PrintN("")
          If user$
            WebPass$=HTTP_CryptedUserPass(user$,pass$)
            HTTP_ChangeWWWAuthenticate(Get,WebPass$)
            If HTTP_ReConnect(Get); neu anfordern
              ReTry=#True
            Else
              PrintN("Can't connect")
            EndIf
          Else
            PrintN("  abort")
          EndIf
        Case 407;Proxy Authentication Required
          PrintN("Proxy Authentication Required:"+HTTP_GetAuthenticateRealm(Get))
          Print("  UserName:"):user$=Input():PrintN("")
          Print("  Password:"):pass$=Input():PrintN("")
          If user$
            ProxyPass$=HTTP_CryptedUserPass(user$,pass$)
            HTTP_ChangeProxyAuthenticate(Get,ProxyPass$)
            If HTTP_ReConnect(Get); neu anfordern
              ReTry=#True
            Else
              PrintN("Can't connect")
            EndIf
          Else
            PrintN("  abort")
          EndIf
        Default
          PrintN("ServerError:")
          PrintN(Str(HTTP_GetError(Get))+" "+HTTP_GetErrorMessage(Get))
      EndSelect
    Until ReTry=#False
    
    PrintN("")
    PrintN("FileSize:"+Str(HTTP_GetFileSize(Get)))
    PrintN("FileDate:"+FormatDate("%dd.%mm.%yyyy %hh:%ii:%ss",HTTP_GetFileDate(Get)))
    
  Else
    PrintN("Cann't connect")
  EndIf
  
  PrintN("")
  PrintN("Press any key")
  Input()
  CloseConsole()
EndIf

End
Last edited by GPI on Sat Dec 11, 2004 4:56 pm, edited 1 time in total.
DEU.exe
User
User
Posts: 13
Joined: Sun Oct 19, 2003 11:21 am
Location: www

Post by DEU.exe »

@GPI: VERY impressive! 8O

...and breathtaking quick.

Is there a possibility downloading to *memory?
pIII@1Ghz, wXP, 512MB, pb3.93full
GPI
PureBasic Expert
PureBasic Expert
Posts: 1394
Joined: Fri Apr 25, 2003 6:41 pm

Post by GPI »

first of all: i change a routine in the code above (HTTP_ReceiveData())

Code: Select all

;please update

Structure HTTPGetId
  url$
  server$
  File$
  FileName$
  Port.l
  wenc$
  ProxyServer$
  ProxyPort.l
  penc$
  cookie$
  location$
  ConnectionID.l
  ErrorMessage$
  FileSize.l
  Date.l
  URLError.l
  Buffer.l
  BufferLength.l
  Realm$ 
  
  InHeader.l
  FirstLine.l
  currentline$
  lastchar.l
  eol.l
  Received.l
  Typ$
  
  OutBuffer.l
  FileHandle.API_FileHandle
EndStructure

;new procedure

Procedure HTTP_ReceiveDataMemory(*Handle.HTTPGetId) ; #True=Not ready  #False=AllDatasRecived
  Result=#True
  NCEvent = NetworkClientEvent(*Handle\ConnectionID)
  If NCEvent=2 ; Raw-data
    If *Handle\Buffer=0
      *Handle\Buffer=AllocateMemory(*Handle\BufferLength)
      If *Handle\Buffer=0
        *Handle\URLError=#URLERROR_OutOfMemory
        ProcedureReturn #False
      EndIf
    EndIf
    
    InBuffer=ReceiveNetworkData(*Handle\ConnectionID,*Handle\Buffer,*Handle\BufferLength)
    *start.BYTE=*Handle\Buffer
    If *Handle\InHeader ;{ Header analysieren
      While InBuffer
        char=(*start\b & $FF):InBuffer-1:*start+1
        Select char
          Case #CR;: #crlf$
          Case #LF
            If *Handle\lastchar=#CR; Ende der Zeile entdeckt!
              If *Handle\eol; zum zweiten mal -> nun daten
                *Handle\InHeader=#False
                ;{-Header eingelesen ->Auswerten
                If *Handle\URLError<>200 
                  Result=#False
                Else;OK, datei öffnen
                  If *Handle\FileSize And *Handle\Typ$="GET"
                    *Handle\OutBuffer=AllocateMemory(*Handle\FileSize)
                    If *Handle\OutBuffer=0
                      *Handle\URLError=#URLERROR_OutOfMemory
                      Result=#False
                    EndIf
                  Else
                    Result=#False
                  EndIf
                EndIf
                ;}
                Break
              Else
                *Handle\eol=#True
                Debug *Handle\currentline$
                If *Handle\FirstLine;{-Status-code auslesen
                  *Handle\FirstLine=#False
                  a1=FindString(*Handle\currentline$," ",0)
                  a2=FindString(*Handle\currentline$," ",a1+1):If a2=0:a2=Len(*Handle\currentline$)+1:EndIf
                  ;HTTPVer:Left(*Handle\currentline$,a1-1)
                  *Handle\URLError=Val(Mid(*Handle\currentline$,a1+1,a2-a1-1))
                  *Handle\ErrorMessage$=Right(*Handle\currentline$,Len(*Handle\currentline$)-a2)
                  ;}
                Else;{-Header-Zeile analysieren
                  a=FindString(*Handle\currentline$,":",1)
                  Variable$=Trim(Left(*Handle\currentline$,a-1))
                  Set$=Trim(Right(*Handle\currentline$,Len(*Handle\currentline$)-a))
                  Select UCase(Variable$)
                    Case "PROXY-AUTHENTICATE";{  Realm auslesen
                      a=FindString(Set$,"realm=",0)
                      b=FindString(Set$,#DQUOTE$,a+7)
                      If a And b
                        *Handle\Realm$=Mid(Set$,a+7,b-a-7)
                      EndIf
                      ;}
                    Case "WWW-AUTHENTICATE";{    Realm auslesen
                      a=FindString(Set$,"realm=",0)
                      b=FindString(Set$,#DQUOTE$,a+7)
                      If a And b
                        *Handle\Realm$=Mid(Set$,a+7,b-a-7)
                      EndIf
                      ;}
                    Case "LOCATION":            *Handle\location$=Set$
                    Case "SET-COOKIE";{          Cookie suchen
                      a=FindString(Set$,";",0):If a=0: a=Len(Set$)+1 :EndIf
                      *Handle\cookie$=Left(Set$,a-1)
                      ;}
                    Case "CONTENT-LENGTH":      *Handle\FileSize=Val(Set$)
                    Case "LAST-MODIFIED":       *Handle\Date=HTTP_AnalyseDate(Set$)
                  EndSelect    
                  ;}
                EndIf
                *Handle\currentline$=""
              EndIf
            EndIf
          Default
            *Handle\eol=#False
            *Handle\currentline$+Chr(char)
        EndSelect
        *Handle\lastchar=char
      Wend
      ;}
    EndIf
    If *Handle\InHeader=#False ;{
      If *Handle\Received+InBuffer<=*Handle\FileSize
        CopyMemory(*start,*Handle\OutBuffer+*Handle\Received,InBuffer)
      EndIf
      *Handle\Received+InBuffer
      If *Handle\Received=*Handle\FileSize
        Result=#False;fertig
        API_CloseFile(*Handle\FileHandle)
      EndIf
      ;}
    EndIf
  EndIf
  
  ProcedureReturn Result
EndProcedure

;-new example
If InitNetwork()
  
  OpenConsole()
  PrintN("***Getting file information***")
  If HTTP_UrlInfo(Info.HTTPGetId,url,WebPass$,ProxyServer$,ProxyPort,ProxyPass$,0)
    PrintN("connect...")
    While HTTP_ReceiveData(Info); wait for header
      Delay(100)
    Wend
    HTTP_CloseURL(Info)
    Select HTTP_GetError(Info)
      Case #URLError_OK
        PrintN("FileSize:"+Str(HTTP_GetFileSize(Info)))
        PrintN("FileDate:"+FormatDate("%dd.%mm.%yyyy %hh:%ii:%ss",HTTP_GetFileDate(Info)))
      Case #URLERROR_OutOfMemory:PrintN("Out of memory")
      Case #URLError_CantCreateFile:PrintN("Can't create file")
        ;for all other errors: See download
      Default
        PrintN(Str(HTTP_GetError(Info))+" "+HTTP_GetErrorMessage(Info))
    EndSelect
  EndIf
  
  
  PrintN("")
  PrintN("***Getting file***")
  If HTTP_OpenUrl(Get.HTTPGetId,File$,url,WebPass$,ProxyServer$,ProxyPort,ProxyPass$,0)
    Repeat
      PrintN("Connect...")
      ReTry=#False:Progress=0
      While HTTP_ReceiveDataMemory(Get)
        a=HTTP_GetProgress(Get)
        If a<>Progress
          PrintN("Progress: "+Str(a)+"%")
          Progress=a
        EndIf
        ;if want abort
        ;  http_closeurl(get$)
        ;  printn("Abort")
        ;  break 2
        ;endif
        Delay(100)
      Wend
      HTTP_CloseURL(Get)
      
      Select HTTP_GetError(Get)
        Case #URLError_OK:PrintN("Download complete")
          OutBuffer=HTTP_GetOutBuffer(Get)
          Size=HTTP_GetFileSize(Get)
          Print("save to "+File$+".mem."+GetExtensionPart(File$))
          API_FileCreate(out.API_FileHandle,File$+".mem."+GetExtensionPart(File$))
          API_WriteData(out,OutBuffer,Size)
          API_CloseFile(out)
          freememory(outbuffer)
          PrintN(" done") 
        Case #URLERROR_OutOfMemory:PrintN("Out of memory")
        Case #URLError_CantCreateFile:PrintN("Can't create file")
        Case 301;Moved permanently
          PrintN("Moved permanently:"+HTTP_GetNewLocation(Get))
          HTTP_ChangeURL(Get,HTTP_GetNewLocation(Get))
          If HTTP_ReConnect(Get); neu anfordern
            ReTry=#True
          Else
            PrintN("Can't connect")
          EndIf
        Case 302;Moved temporarily
          PrintN("Moved temporarily:"+HTTP_GetNewLocation(Get))
          HTTP_ChangeURL(Get,HTTP_GetNewLocation(Get))
          If HTTP_ReConnect(Get); neu anfordern
            ReTry=#True
          Else
            PrintN("Can't connect")
          EndIf
        Case 305;Use Proxy
          PrintN("Use Proxy:"+HTTP_GetNewLocation(Get))
        Case 401;unauthorized
          PrintN("Unauthorized:"+HTTP_GetAuthenticateRealm(Get))
          Print("  UserName:"):user$=Input():PrintN("")
          Print("  Password:"):pass$=Input():PrintN("")
          If user$
            WebPass$=HTTP_CryptedUserPass(user$,pass$)
            HTTP_ChangeWWWAuthenticate(Get,WebPass$)
            If HTTP_ReConnect(Get); neu anfordern
              ReTry=#True
            Else
              PrintN("Can't connect")
            EndIf
          Else
            PrintN("  abort")
          EndIf
        Case 407;Proxy Authentication Required
          PrintN("Proxy Authentication Required:"+HTTP_GetAuthenticateRealm(Get))
          Print("  UserName:"):user$=Input():PrintN("")
          Print("  Password:"):pass$=Input():PrintN("")
          If user$
            ProxyPass$=HTTP_CryptedUserPass(user$,pass$)
            HTTP_ChangeProxyAuthenticate(Get,ProxyPass$)
            If HTTP_ReConnect(Get); neu anfordern
              ReTry=#True
            Else
              PrintN("Can't connect")
            EndIf
          Else
            PrintN("  abort")
          EndIf
        Default
          PrintN("ServerError:")
          PrintN(Str(HTTP_GetError(Get))+" "+HTTP_GetErrorMessage(Get))
      EndSelect
    Until ReTry=#False
    
    PrintN("")
    PrintN("FileSize:"+Str(HTTP_GetFileSize(Get)))
    PrintN("FileDate:"+FormatDate("%dd.%mm.%yyyy %hh:%ii:%ss",HTTP_GetFileDate(Get)))
    
  Else
    PrintN("Cann't connect")
  EndIf
  
  PrintN("")
  PrintN("Press any key")
  Input()
  CloseConsole()
EndIf

End
And don't forget to free the output-buffer!
User avatar
NoahPhense
Addict
Addict
Posts: 1999
Joined: Thu Oct 16, 2003 8:30 pm
Location: North Florida

Post by NoahPhense »

@GPI..

So, this second code-post, can be added to the first code-post above it.
And there will be no conflicts?

I have your old lib.. so what you're saying is to use this code, and NOT
the Lib right?

- np
GPI
PureBasic Expert
PureBasic Expert
Posts: 1394
Joined: Fri Apr 25, 2003 6:41 pm

Post by GPI »

NoahPhense wrote:@GPI..

So, this second code-post, can be added to the first code-post above it.
And there will be no conflicts?
Replace the structure and the example. paste the two procedures

>I have your old lib..

the GetURL-Lib is from Num3, not from me.

> so what you're saying is to use this code, and NOT
the Lib right?

When the lib is based on the source in the thread-start-posting: Yes, you should not use the lib.

btw.: Thanks to num3, because with his lib, i can't write my code.


- np[/quote]
FloHimself
Enthusiast
Enthusiast
Posts: 229
Joined: Wed May 14, 2003 3:38 pm
Location: Lüneburg - Germany

Post by FloHimself »

here it's working with HTTP-Proxy!

EDIT: with password ;)
DEU.exe
User
User
Posts: 13
Joined: Sun Oct 19, 2003 11:21 am
Location: www

Post by DEU.exe »

Maybe I'm blind, but... where's the HTTP_GetOutBuffer() Procedure?

Error in line 637 (not a function)...
pIII@1Ghz, wXP, 512MB, pb3.93full
GPI
PureBasic Expert
PureBasic Expert
Posts: 1394
Joined: Fri Apr 25, 2003 6:41 pm

Post by GPI »

ups.

Code: Select all

Procedure HTTP_GetOutBuffer(*Handle.HTTPGetId); File must be downloaded with HTTP_ReceiveDataMemory()
  ProcedureReturn *Handle\OutBuffer
EndProcedure
Num3
PureBasic Expert
PureBasic Expert
Posts: 2810
Joined: Fri Apr 25, 2003 4:51 pm
Location: Portugal, Lisbon
Contact:

Post by Num3 »

NEWS!!!

This library has been rebuilt -> HTTP library
GPI
PureBasic Expert
PureBasic Expert
Posts: 1394
Joined: Fri Apr 25, 2003 6:41 pm

Post by GPI »

Num3 wrote:NEWS!!!

This library has been rebuilt -> HTTP library
Source?
Dare2
Moderator
Moderator
Posts: 3321
Joined: Sat Dec 27, 2003 3:55 am
Location: Great Southern Land

Post by Dare2 »

@GPI

Thanks for the code there. Appreciated. :)
@}--`--,-- A rose by any other name ..
Post Reply