Hashtopolis Forum
Slow retrieval of the hashlist. - Printable Version

+- Hashtopolis Forum (https://hashtopolis.org)
+-- Forum: Support (https://hashtopolis.org/forum-1.html)
+--- Forum: Problems (https://hashtopolis.org/forum-2.html)
+--- Thread: Slow retrieval of the hashlist. (/thread-1791.html)



Slow retrieval of the hashlist. - F8k4X - 05-26-2022

Hello.

After uploading dozens of hash files - 5,000,000 lines each, 'Hash.hash' column has 132,650,000 rows. Currently, the mysql server takes about 50 seconds to process a batch of 50,000 hashes from the database, which is about 80 minutes before the client downloads the hash list. Processing query with using index takes about 2 seconds, which would reduce the download time to about 3 minutes. Below are the results from the slow-queries log of the mysql server.

Code:
# Query_time: 51.265938  Lock_time: 0.000003 Rows_sent: 50000  Rows_examined: 1350000
SET timestamp=1653555672;
SELECT hashId, hashlistId, hash, salt, plaintext, timeCracked, chunkId, isCracked, crackPos FROM Hash WHERE hashlistId='30' AND isCracked='0' ORDER BY Hash.hashId ASC LIMIT 1300000,50000;

# Query_time: 2.083822  Lock_time: 0.000003 Rows_sent: 50000  Rows_examined: 1400000
SET timestamp=1653555792;
SELECT hashId, hashlistId, hash, salt, plaintext, timeCracked, chunkId, isCracked, crackPos FROM Hash USE INDEX (hashlistId) WHERE hashlistId='30' AND isCracked='0' ORDER BY Hash.hashId ASC LIMIT 1350000,50000;

Could I find out if it is possible to modify the query to search by index of the table?

Hashtopolis: 0.12.0
Client: 0.6.0.10


RE: Slow retrieval of the hashlist. - s3in!c - 09-14-2022

Hashtopolis in general is not meant normally to deal with so many multi-million Hashlists.

But your hint may be a good idea to add. Can you please open a feature request on Github, so we can put it on our to do list?