Page 1 of 4

Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 1:36 pm
by TheAutomator
I started learning PureBasic because i wanted to program in a readable and fairly easy to use programming language (unlike C++).

Pro's:
* PureBasic has no dependency's.
* It compiles to native executables and does not need any dll's.
* It's very fast.
* It supports 64 bit.
* Has some handy functionality build in.

As many basic programmers i came from VB6 and discovered PureBasic.
The main difference tho, is that PureBasic has it's limits compared to VB..

Cons:
* no variant types possible.
* functions can only return basic types like numbers or strings.
* no support for OOP.
* lists can only be nested with dirty looking hacks that involve pointers and allocating memory.
* ActiveX COM is 'somewhat supported' if you know what you're doing.

The main goal of this project is to build a VBScript like interpreter that works with script files,
the status of this project:

- My interpreter has to read a file (given as a parameter) -> DONE!
- My interpreter has to tokenize the contents of that file -> DONE!
- My interpreter has to check for lexical errors -> DONE!

- My interpreter has to handle variants -> WORKAROUND = use a structure that holds every type like this:

Code: Select all

structure token
   type.i ; if the type would change then i have to empty and fill one of those guys here:
   number.d
   string.s
endstructure
Here i start to get in trouble:

- My interpreter has to convert it's input to an abstract syntax tree -> PROBLEM

In VB i would create classes for every action and nest those classes in a tree, here is a peace of code from VB to show what i mean:
Using the GoldParser back in the days: http://goldparser.org

Code: Select all

Public Function CreateSimpleIfStm(IfClause As Object, ThenClause As Object, Optional ElseClause As Object = Nothing) As SimpleIfStm
    Dim Obj As New SimpleIfStm
    
    Set Obj.IfClause = IfClause
    Set Obj.ThenClause = ThenClause
    Set Obj.ElseClause = ElseClause
    
    Set CreateSimpleIfStm = Obj
End Function

Public Function CreateSimpleWhileStm(WhileClause As Object, DoClause As Object) As SimpleWhileStm
    Dim Obj As New SimpleWhileStm
    
    Set Obj.WhileClause = WhileClause
    Set Obj.DoClause = DoClause
        
    Set CreateSimpleWhileStm = Obj
End Function

Public Function CreateSimpleStmList(CurrentStm As Object, Optional NextStm As Object = Nothing) As SimpleStmList
    Dim Obj As New SimpleStmList
    
    Set Obj.CurrentStm = CurrentStm
    ' currentstm is nothing
    If Not NextStm Is Nothing Then
        Set Obj.NextStm = NextStm
    End If
            
    Set CreateSimpleStmList = Obj
End Function
That's not possible in PureBasic because OOP is not supported.
Okay i can deal with that, what about instead of objects i use arrays or lists and nest them so i can traverse the tree later?
also not possible! Heck it's not even possible to nest lists with different types or add different types to lists.. :?
What about function blocks? Filling nested arrays in our script language and store them?

So yeah... now I'm stuck and i have no idea how anyone else would solve this problem.
I know there are a few guys on this forum that are interested in making interpreters or programming languages (or already did that).

Can anyone with experience show me the way / share some example code?

regards, TheAutomator.

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 2:20 pm
by Tenaja
Just get comfortable with pointers. Once you are comfortable with them, those "dirty hacks" just become "simple solution".

Have you searched for your favorite of one of the dozens of oop implementations on here?

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 2:54 pm
by STARGÅTE
I posted my small example already here.
You can call these "NewObject" procedures during running your parser and creating the tree.
But in my opinion, while you "started learning PureBasic", I would not recommend you to write an interpreter as a first first project in Pure Basic. It's not an easy thing.

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 4:32 pm
by TheAutomator
Tenaja wrote:Just get comfortable with pointers. Once you are comfortable with them, those "dirty hacks" just become "simple solution".

Have you searched for your favorite of one of the dozens of oop implementations on here?
Yeah pointers are quite new to me, it's like working with a black box and not really knowing what I'm doing in it exactly,
what dangerous stuff I'm doing with memory and pointers.. Memory leaks for example, or forgetting something important like freeing memory the right way.

Dozens of oop implementations? of what?

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 4:40 pm
by TheAutomator
STARGÅTE wrote:I posted my small example already here.
You can call these "NewObject" procedures during running your parser and creating the tree.
But in my opinion, while you "started learning PureBasic", I would not recommend you to write an interpreter as a first first project in Pure Basic. It's not an easy thing.
Yes and i appreciate the help, but if feels a bit like messing with the PB language instead of cleanly using it the right way?
There should be a reason you can't directly do that stuff in PB and instead need to keep track of what you're doing right? :
keep track of low level memory stuff and not crashing your pc,
a lot of workaround code to do one simple thing..

building an interpreter on itself is already advanced, and now it's also messing with low level pointer memory workaround functions on top.. :?
I wonder if PB actually is he right language to do this stuff with.
But it's the best software that is available for my needs atm:

basic understandable syntax, fast, compact native executable, ...

what i would love to see is VB with the power of PB.
It doesn't seem to exist, PB is the closest i can get / find, and i love the language, don't get me wrong!

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 6:00 pm
by Tenaja
If you search this forum, many people have shared their version of oop implementations.

What languages have you used that handles nested lists without pointers? I know I haven't used many of the newer ones, but it seems like most I have used have used pointers for this.

Once you get a handle on them, pointers become second nature. You make yourself some routines (functions or macros) to insert and add lists, and then the rest of your ast will fall in place.

I was actually quite surprised at how simple (short, in code line count) an interpreter or compiler can be, once I started studying them.

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 6:13 pm
by TheAutomator
Tenaja wrote:If you search this forum, many people have shared their version of oop implementations.

What languages have you used that handles nested lists without pointers? I know I haven't used many of the newer ones, but it seems like most I have used have used pointers for this.

Once you get a handle on them, pointers become second nature. You make yourself some routines (functions or macros) to insert and add lists, and then the rest of your ast will fall in place.

I was actually quite surprised at how simple (short, in code line count) an interpreter or compiler can be, once I started studying them.
"What languages have you used that handles nested lists without pointers?"
well, have a look at my post, VB6 was a handy one, especially with the gold parser (see link).
as a kid i even made a small language in vbscript hehe :lol:
Lua can do that, AutoIt code can do that with arrays, Python, Ruby, ...

I'm an incredible fan of BASIC languages because they are simple, readable and don't require to almost write assembly.
C++ is one of the worst syntaxes in my opinion:

Code: Select all

#define GET_VAL( val, type ) \
    {                                                   \
        ASSERT( ( pIP + sizeof(type) ) <= pMethodEnd ); \
        val = ( *((type *&)(pIP))++ );                  \
    }
compared to VB:

Code: Select all

SUB MATH
   A = 15
   B = 1 + A
   MSGBOX B
END SUB
PB seems to hover somewhere in the middle :lol: .

i'm gonna search on OOP in the forums here meanwhile :)

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 7:01 pm
by mk-soft
Gave up too quickly?

It takes a while to understand the change from VB to Purebasic.

Purebasic offers a lot. Have a look at the help.
The handling of memory is more like 'C' and you can play with pointers very well. (Better as VB)

Purebasic supports the basics of object oriented programming, but with the same effort as 'C' (not C++). So a lot of self-declaration and programming (if you like)

Why write an interpreter yourself! If it is for Windows, you can use the ActiveScript control. This supports VB-Script and JScript.

On Windows the variable type Variant is also supported by API.

It will take some time to see what is really possible with Purebasic. And that is in my opinion very much.

Interpreter
Link: Module ActiveScript for VB-Script with PB-Runtime Variables

OOP and More
Show signature

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 8:05 pm
by TheAutomator
Okay, I'll deepen myself more in pointers.
Didn't know it was totally okay to use them that way in PB, i'm afraid of not fully understanding it and crashing my pc
(or worse! not knowing something is going horribly wrong!).

I read the help files but i must admit that the small examples written there sometimes give me more questions then answers..
For example, i read the part about "structureunion" 3 times and still don't have a clue what it's used for and why :(

why i'm writing my own language?
Simple, it's a project, something you can design and finetune yourself, make up your own syntax.
It's exiting! Where is the fun at if someone else did it for you and you can't change anything about how it works.
Why do people build there own computers or OS if they can just buy one and deal with it even if they don't like the design?

You can look at it as a bucket-list item :)

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 8:37 pm
by STARGÅTE
TheAutomator wrote:For example, i read the part about "structureunion" 3 times and still don't have a clue what it's used for and why
StructureUnion allows the access to the same memory position but different handling of the conent:
You can write or read different content (floats, integers, strings) with the same variable:

Code: Select all

Structure SomeStructure
	StructureUnion
		Quad.q      ; 8 bytes
		Char.c[4]   ; 4 characters, each 2 bytes
		String.s{4} ; 4 characters, each 2 bytes
	EndStructureUnion
EndStructure

Define SomeStructure.SomeStructure
SomeStructure\String = "TEST"

Debug SomeStructure\String
Debug SomeStructure\Quad

SomeStructure\Char[0] - 18 ; decrease the first character by 18: T -> B
SomeStructure\Char[3] + 1 ; increase the last character by 1: T -> U

Debug SomeStructure\Quad
Debug SomeStructure\String
TheAutomator wrote:why i'm writing my own language?
Simple, it's a project, something you can design and finetune yourself, make up your own syntax.
That's fine, but not if you are a newcomer in a language right?
TheAutomator wrote:Yes and i appreciate the help, but if feels a bit like messing with the PB language instead of cleanly using it the right way?
I don't understand this answer. What "right way"?
In Pure Basic you have to define every content by a variable type (float, integer, string) or a structure.
You can apply them to a single variable, array, list or map. You can also nest them, but with a predefined structure.
If you want to use tree-like hierarchies with different contents in Pure Basic (directly) you have to use pointers and you have to organize your content by yourself. Thats a "right why". But you can also use standards like XML or JSON in Pure Basic to store this content.

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 10:11 pm
by TheAutomator
"but not if you are a newcomer in a language right?"

yeah, i'm a bit inpatient sometimes (guilty), the main reason i use PB is for that atm.

"I don't understand this answer. What "right way"?"

It's me being a newbie, pointers feel like low lever stuff, and PureBasic want you to use the same type all the way trough, i thought changing it with memory functions was a bit dangerous and not intended. In the help files you read "only for advances users!!!" and "warning, can cause memory leaks!!!" a lot.

"StructureUnion allows the access to the same memory position but different handling of the conent"
Okay, i get it now :D
so it's basically handeling a variable in multiple ways, in this case like a string and a char and a quad on the fly :)
Wouldn't that be a great solution for variants?

like this:

Code: Select all

Structure variant_type
   type.i
   StructureUnion
      string.s
      quad.q
   EndStructureUnion
EndStructure
instead of:

Code: Select all

Structure variant_type
   type.i
   string.s
   quad.q
EndStructure

Re: Crafting an interpreter in PureBasic

Posted: Mon Dec 07, 2020 11:17 pm
by mk-soft
It used to work in the past, but it led to the automatic release of memory, which is not clear at runtime whether it is a value or a string (i.e. the pointer to a string). This then led to memory protection violation

This was therefore no longer allowed.

Solution:

Code: Select all

Structure variant_type
  type.i
  StructureUnion
    bVal.b
    wVal.w
    iVal.i
    lVal.l
    qVal.q
    fltVal.f
    dblVal.d
  EndStructureUnion
  sVal.s
EndStructure

*mem1.variant_type = AllocateStructure(variant_type)
*mem1\type = #PB_String ; I use a read constant from PB for Type
*mem1\sVal = "Hello World!"
Debug *mem1\sVal

*mem2.variant_type = AllocateStructure(variant_type)
*mem2\type = #PB_Long ; I use a read constant from PB for Type
*mem2\lVal = 100
Debug *mem2\lVal

; Memory for string is automatic released
FreeStructure(*mem1)
; No Memory for string used
FreeStructure(*mem2)

Re: Crafting an interpreter in PureBasic

Posted: Tue Dec 08, 2020 10:39 pm
by Olli
I imagine a structure like this

Code: Select all

#Undef = $0
#Integ = $001
#Strin = $102
#Float = $003

#aList = $200
#Table = $400 ; (as an array)
#Bundl = $B00


Structure Variant
   Cfg.I
   Dta.I
EndStructure

Procedure VariantCreate(Cfg.I = #Undef)
   Define *This.Variant
   *This = AllocateMemory(SizeOf(Variant) )
   *This\Cfg = Cfg
   ProcedureReturn *This
EndProcedure

Procedure VariantDestroy(*This.Variant)
   Define I
   Define CpnQty
   If *This\Cfg & $100
      CpnQty = *This\Cfg & $7FFFF000
      CpnQty >> 12
      If CpnQty = 1
         If *This\Cfg & $400
            VariantDestroy(*This\Dta)
         Else
            FreeMemory(*This\Dta)
         Endif
      Else
         For I = 1 To CpnQty
            *That = PeekI(*This\Dta + SizeOf(Integer) * (I - 1) )
            If *This\Cfg & $400
               VariantDestroy(*That) ; recursivity
            Else
               FreeMemory(*That)
            EndIf
         Next
         FreeMemory(*This\Dta)
      EndIf
   EndIf
   FreeMemory(*This)
EndProcedure
Cfg = data ConFiGuration
Dta = DaTA

Cfg settings bits (31 bits used, to be retro-compatible) :

Bit 0 to 7 : type of variable (256 types available)
Bit 8 : 1 if requires a sub-memory, else 0
Bit 9 : 1 if the composition of sub-memory is variable, else 0
Bit 10 : 1 if the sub-memory contains its own sub-memory too, else 0
Bit 11 : 1 if the components size is variable, else 0
Bit 12 to 30 : quantity of components in the sub-memory. (from 0 to 524287)

Simplest type creatings proceed like this :

Code: Select all

Procedure IntegerCreate(Dta.I = 0)
    Define *This.Variant = VariantCreate($1000 |  #Integ)
    *This\Dta = Dta
   ProcedureReturn *This
EndProcedure

Procedure StringCreate(Dta.S = "")
   Define *This.Variant = VariantCreate($1000 | #Strin)
   *This\Dta = AllocateMemory(StringByteLength(Dta + "Z") )
   PokeS(*This\Dta, Dta)
EndProcedure
Remark :


a. memory overview when there is only 1 sub-memory :

[variant] ---> [external datas]

b. memory overview when there are more than 1 sub-memory :

[variant] ---> [addresses table] ---> [variant / external datas]

Re: Crafting an interpreter in PureBasic

Posted: Wed Dec 09, 2020 3:41 pm
by Thorium
TheAutomator wrote: * no support for OOP.
Not con for me. Actually one of the few reasons PureBasic is still relevant to me.

Since i am working on a fairly big game project in UE4 and C++ for the last few years, i can really say: OOP introduces more problems than it solves. I spend so much time thinking how to structure my code, and what needs to hold references to what. It's redicilous. In many cases you end up making manager classes the whole purpose of these is to manage the objects underneath them. Still you got many cross references and they defeat the main purpose of OOP. And if your object hirarchy doesnt work out at the end, have fun to refactor everything. The funny part is the bigger your project the worse your problems get later on if you made a mistake at the beginning. The very thing OOP should be better at: Managing big projects.
I use it because if have to. UE4 relies heavily on OOP concepts and it would be an even bigger nightmare to try and force it into an imperitive style. Actually it wouldnt even possible without scraping the whole editor.

I encountered absolutly no problem i couldnt solve easily without OOP. However i encountered a lot of problems caused by flaws in my OOP design. And quite big ones, which required major rethinking and refactoring. You may think, well you need to prepare better beforhand and lay out your structure and design properly. But thats not realistic. Developing a project is often times a very organic process. Things dont work out as you imagined. Great ideas come when your project is actually used and you need to implement them. OOP forces you into a structure, which is just not improving the work flow at all.

Re: Crafting an interpreter in PureBasic

Posted: Thu Dec 10, 2020 6:01 am
by Rinzwind
The OOP situation is a bid sad, because just a tiny few things are missing to make it very usable for the majority of OOP use cases (and still makes it an optional thing): improve the structure type to allow it to contain function pointers and an automatic self pointer and custom default values. Voila. No more need for tedious, error prone boiler-plate code to create instances of both data and functionality on that data. OOP can have multiple meanings. Most just want a convenient way to structure data and its behaviour. It's also a replacement and enhancement for the wonky Module thing. Makes multi-instancing of windows a breeze too. Please. No one is asking for c++ levels of cumbersomeness and possibilities. If Fred adds that tiny Structure feature we get encapsulation, inheritance and polymorphism, all optional, which can be immensely useful to simplify coding. The OOP attempts here (not by the least of PB users, aka proficient in the language and supporters) make me cringe, because it could be so much more elegant when PB adds a tiny bit of syntax.