Chromium WebGadget v4

Developed or developing a new product in PureBasic? Tell the world about it.
User avatar
skinkairewalker
Enthusiast
Enthusiast
Posts: 627
Joined: Fri Dec 04, 2015 9:26 pm

Re: Chromium Browser WebGadget

Post by skinkairewalker »

Hello, I found the project interesting, but I have some doubts...

*can I export as a standalone executable?
*Is it mandatory to install external dependencies in order to work?
*consume a lot of ram?
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium Browser WebGadget

Post by JHPJHP »

Hi skinkairewalker,
skinkairewalker wrote:*can I export as a standalone executable?
Free executables (x86/x64) can be downloaded from My PureBasic Stuff; see FREE STUFF.
skinkairewalker wrote:*Is it mandatory to install external dependencies in order to work?
No external dependencies other then a supported browser...
The WebGadget currently works with Edge and Chrome, but other Chromium based browsers could be added.
The Standalone application currently works with Edge, Chrome, Opera, Vivaldi, Brave, Epic, but other Chromium based browsers could be added.
skinkairewalker wrote:*consume a lot of ram?
No, but test the free executables for yourself.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium Browser WebGadget

Post by JHPJHP »

Updated:
- added some error checking
- small improvement to Headless mode

Error Checking
Previously, the scripts where JSON is parsed had the potential to cause a crash, this has now been fixed.
Last edited by JHPJHP on Wed Jul 13, 2022 2:12 am, edited 1 time in total.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated:
- improved CDP Control Panel
- improved CBWG DevTools examples
- fixed a couple things

CDP Control Panel
1. Run the example CDP_Control_Panel.pb.
2. Press the Select Browser button.
3. Select an available browser.
4. Press the Listen Port button.
5. Press the Connect DevTools button.
6. Right-mouse-click the Command List.
7. Select the Load Command List... option.
8. Select the file compile.json.
9. Right-mouse-click the Command List.
10. Select the Send All option.
11. On the webpage enter jhpjhp in the Search field.

- Navigates to the PureBasic forum.
- Compiles some user-defined JavaScript (includes an event listener)
- Includes the compiled JavaScript into the PureBasic webpage.
- When jhpjhp is entered in the search field all the forum titles are changed to JHPJHP.

CBWG DevTools
1. Run the example CB_WebGadget_Detail.pb.
2. Select the Page.navigate ComboBox item.
3. Press the Send Command button.
4. Select the DOM.getNodeForLocation ComboBox item.
5. Manually scroll the webpage to any location.
6. Put the mouse-pointer over any hyperlink.
7. Press the Insert key to update parameters.
8. Press the Send Command button.

- Returns the Layout Metrics from the webpage.
- Finds the X/Y scroll position from the Layout Metrics.
- Returns the X/Y mouse location (Insert key Procedure).
- Calculates the combined scroll position and mouse location.
- Returns the Node ID under the mouse pointer using the calculation.
- Using the returned Node ID extracts the outer HTML associated to the node.

Image
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated:
- improved CDP Control Panel
-- added DevTools Protocol help / filter (F1, F2, F3, Esc)
- minor improvements to DevTools examples

Early on there was some interest regarding the Chromium WebGadget, but at that time only 20 plus hours were invested, and most of the WebGadget was experimental. Very little of that earlier version can be found in the latest examples.

Suffice to say, if you're doing any programming that requires access to elements of a webpage, you should be familiar with the DevTools Protocol.

Currently, 400 plus hours have been spent writing the Chromium WebGadget, and at least half of that time was dedicated to understanding and incorporating the DevTools Protocol into the CB WebGadget and standalone Control Panel.

NB*: See previous post for some basic instructions.
Last edited by JHPJHP on Tue Nov 22, 2022 10:52 pm, edited 2 times in total.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated (CDP Contol Panel)
- added support for the Maxthon browser
- various minor improvements
Last edited by JHPJHP on Wed Nov 02, 2022 12:58 am, edited 1 time in total.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated:
- added URL blocking support
- various minor improvements
- squashed a small bug

URL Blocking Support
Created a JSON file to demonstrate how easy it is to block URLs using the DevTools Protocol.
Includes event driven script prompting the user to decline or allow access to the URL.

-------------------------------------------------------

Updated:
- extended URL blocking support to return document body
- added URL blocking support to CB_WebGadget_Detail.pb
- squashed a bug

CB_WebGadget_Detail.pb
The document body is now returned during the Fetch request allowing for editing before being passed to the browser.

1. Execute the Fetch.enable command.
2. Execute the Page.navigate command.
3. If the page is allowed to load, the word Windows is substituted with JHPJHP.

Because the document body was larger then 65536 bytes, this was the first time I was able to test my WebSocket (send frame) fragmentation script; hence the squashed bug.
Last edited by JHPJHP on Thu Jul 21, 2022 9:00 pm, edited 1 time in total.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated:
- improved page connections
- various minor improvements
- better error checking

If you're using the framework or the standalone tool this is a must download as it provides better support for page connections.

DevTools Examples
Previously it was the programmers responsibility to track when a new tab or window was opened and handle accordingly.
Event script has been added to stop new tabs and windows from opening, redirecting the current page to the selected URL.

CDP Control Panel (standalone tool)
Added a window listing multiple page connections.
Pressing the Port Listening button will either populate the Page ID field immediately (single connection), or open the Page Data window when multiple connections are detected.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated:
- added new DevTools example
- couple minor improvements

CB_WebGadget_ChartJS.pb
Line Chart Stacked was copied from the ChartJS Chromium example, and with a few additional lines of code the DevTools Protocol was added. This provided PureBasic the ability to execute JavaScript and manipulate the line chart; see image below.

I was lazy using only the PureBasic ButtonGadget, but the example does include everything needed to create interactive charts.

Using script from the nine charts included in the original Chromium example, combined with the DevTools script included in the new example, should provide enough information to create any of the ChartJS samples with complete JavaScript support.

Something Fun To try (remote controlling a WebGadget)
Run in parallel the CDP Control Panel application. Ignore the Select Browser button and only press the Listen Port button. After establishing a DevTools WebSocket connection to the ChartJS example, any commands sent from the Control Panel will be executed in the WebGadget.

Image
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Updated:
- improved Page handling
- improved DevTools help
- added browser check
- modified readme.txt

Page Handling (DevTools examples)
Noticed that some pages were not being redirected due to an empty returned URL parameter; Target.targetInfoChanged event was added.

DevTools Help (standalone CDP Control Panel)
The Protocol is now filtered on the ComboBoxGadget Changed event.
The filter button now returns the Method, provided both Domain and Command have been selected.
Last edited by JHPJHP on Mon Jul 25, 2022 3:17 am, edited 2 times in total.
User avatar
skinkairewalker
Enthusiast
Enthusiast
Posts: 627
Joined: Fri Dec 04, 2015 9:26 pm

Re: Chromium WebGadget

Post by skinkairewalker »

it would be wonderful, if you could port Chromium WebGadget to Linux & Mac too...
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Hi skinkairewalker,
skinkairewalker wrote:it would be wonderful, if you could port Chromium WebGadget to Linux & Mac too...
Sorry, it's not something I'm planning, especially with the poor forum response. But, all of the CDP Procedures should be cross-platform...
So, I am considering a limited cross-platform tool with similar functionality to the CDP Control Panel.

----------------------------------

Updated:
- couple minor improvements
- rearranged some of the code
Last edited by JHPJHP on Fri Jul 29, 2022 4:59 pm, edited 1 time in total.
zikitrake
Addict
Addict
Posts: 834
Joined: Thu Mar 25, 2004 2:15 pm
Location: Spain

Re: Chromium WebGadget

Post by zikitrake »

I currently use PB.Ex.Webgadget (thank you very much, @RSBASIC) to automate data entry and extraction on my work intranet.

I've been following this project for months and I think it's time to convert my small automators and not depend on CEF Sharp / .NET libraries.

Back from holydays I hope to be able to do the conversion without too much trouble :)

In short, thank you very much for your hard work.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: Chromium WebGadget

Post by JHPJHP »

Hi zikitrake,

I'm glad you're taking a second look at the Chromium WebGadget, you were one of the first people to give support and provide feedback.

Please download the latest version as some Procedure names were recently changed.

---------------------------------------------------

Updated:
- name changes
- small improvements
- removed redundant code
Last edited by JHPJHP on Tue Jul 26, 2022 6:11 am, edited 1 time in total.
zikitrake
Addict
Addict
Posts: 834
Joined: Thu Mar 25, 2004 2:15 pm
Location: Spain

Re: Chromium WebGadget

Post by zikitrake »

Thank you. Done!

I've done a little test and for now I pause with Delay() to make sure that the loading of a page is finished but of course, it doesn't give me much security.

Is there any way to check that a page has actually finished loading?

For example, in PB.Ex Webgadget() there is the option GetWebGadgetExAttribute(ID, #PB_Web_Busy, @ErrorOutput$) to ensure this.

My current procedure:

Code: Select all

Procedure pbCBWEB_PageNavigation(dataUrl.s, waitFullLoad.b = #True)
  Protected SendData.SEND_DATA
  SendData\EventID = 20300
  SendData\Method = "Page.navigate"
  SendData\Params = #DQUOTE$ + "url" + #DQUOTE$ + ":" + #DQUOTE$ + dataUrl + #DQUOTE$
  
  If Not CB_WebSocket_Send(cbWEBConnection, @SendData)
    Debug "Error Sending Frame: " + Str(SendData\EventID) + " in pbCBWEB_PageNavigation(" + dataUrl + ")"
  EndIf
EndProcedure
I have added the waitFullLoad parameter, but I have no idea how to continue :lol:
Locked