Search found 48 matches

by Sooraa
Fri Jun 04, 2021 8:16 am
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

That would bee too nice. Sometimes it's just a matter of beeing aware, what triggers the way we do something: The success of Kmer processing suffers under the reading errors which come out of the analog data capture process. So already the source data are "fuzzy". You find one and the same...
by Sooraa
Thu Jun 03, 2021 1:24 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Wilbert sorry, but there are so many versions, formats, precisions, coverages in different databases and different stati.... The "N"s come from another maskedRepeater and mean also repeats. For our test purposes we can simply scratch them out. To your question: To me, optimizing for 4.7 M...
by Sooraa
Wed Jun 02, 2021 6:56 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

I'm a bit shy to post big data via the board, but to answer your question "... every Kmer... here a little output from the beginning of the output for elegans: I just store K and count. The index is taken internally to retrieve the DNA string in the length of K during output of this test. Just ...
by Sooraa
Wed Jun 02, 2021 6:22 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

The most elegant download site is: https://www.ncbi.nlm.nih.gov/datasets/genomes/?acc=GCA_004526295.1 The download is selfexplanatory. The only thing you should do is to set the file content into all upper case and remove five textual separators ">CP038189.1 ...." Or: If you can provide a ...
by Sooraa
Wed Jun 02, 2021 3:19 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

There might be a slight misunderstanding: At the beginning I described here in the board my idea to create a "Kmer-counter" for the desctop/WS arena under OSX/Windows/Linux. State of the art are solutions with 512 GB memory or Cloud applications. I described that I/we are in the "feas...
by Sooraa
Mon May 31, 2021 4:11 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

Here my status with handling Kmers of DNA: Last we had the tests with the genom of C. ecoli (4,7 Mio nucleos) My testbeds: 4 Haswell-cores, 8 MB L3 16 GB Mem WIN7-64 PB5.43-64 in Ascii Mode only 1 CPU enabled, HT switched off Kmer 1 till 31 Duration: 49 sec. w/o file output of the results 100 sec. w...
by Sooraa
Thu May 27, 2021 4:00 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

I'm back from a few days at the north coast after the repeated shut-downs here.
Still working to present results.
by Sooraa
Thu May 20, 2021 1:47 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Idle

I just saw your pm. Probably this reduces the # of my questions.

I'll come back asap.
by Sooraa
Thu May 20, 2021 1:33 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Idle I don't think that I understood everything from your test. 102 sec for 4,641,652 items (sounds for the ecoli dataset) sounds well on the first view. What does "17=5 memsize mb 0.00 28.75" mean? 17 sounds for k What is the 5? What is "mb 0.00"? What is 28.5? How many threads...
by Sooraa
Wed May 19, 2021 6:47 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

Gent's, you are smart enough to detect my fault:
DNAMAP()\K_Count + 1
comes three lines later than shown...
by Sooraa
Wed May 19, 2021 6:43 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Idle, @Wilbert, it's good to hear that you stick with the problem. Wilbert, to clarify it perhaps a bit more and to show, where I am (pseudo code) : Structure KmerCombo ; it's 2*32 = 64 bits StructureUnion KmerQuad.q DNAindex.l K_Count.l EndStructureUnion EndStructure NewMap KmerMap.Kstruc(15000000...
by Sooraa
Wed May 19, 2021 4:00 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Idle Yeah, any time also general. Perhaps you misunderstood the intend to my last post: Since it became more and more quiet in the topic, I made a proposal to the state of affairs. If you still see chances for a new approach in this matter, you are more than welcome. Your statements to solve the de...
by Sooraa
Wed May 19, 2021 8:48 am
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Idle, @Wilbert I thinks it's time for a break in our efforts to tame the big-data-beast "K-mering" of genomes. I thank you both for your engagement so far. You also helped to detect the limits. What is learned so far? - I will fall back to the approach "change memory consumtion into ...
by Sooraa
Mon May 17, 2021 8:34 pm
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Wilbert Thanks for the link. In fact, I jumped into SSE4.2 through Helle's first 64bit-Ascii-SSE4.2 search function and started back then. :) I created several versions with PCMP*STR* and became familiar with this instructions. But as said, my current try to transform the Equal-ordered Index-versio...
by Sooraa
Mon May 17, 2021 9:53 am
Forum: Assembly Programming
Topic: A small procedure asm
Replies: 125
Views: 38615

Re: A small procedure asm

@Wilbert FYI: In my try to use (3/1c-latency) "popcnt" with 1 addl. clock by using "pcmpistrm" shows the same obstacles as with Google's bigdata n-gram project https://books.google.com/ngrams/info# and 50 TB-data under https://storage.googleapis.com/books/ngrams/books/datasetsv3....