Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Slow retrieval of the hashlist.
#1
Hello.

After uploading dozens of hash files - 5,000,000 lines each, 'Hash.hash' column has 132,650,000 rows. Currently, the mysql server takes about 50 seconds to process a batch of 50,000 hashes from the database, which is about 80 minutes before the client downloads the hash list. Processing query with using index takes about 2 seconds, which would reduce the download time to about 3 minutes. Below are the results from the slow-queries log of the mysql server.

Code:
# Query_time: 51.265938  Lock_time: 0.000003 Rows_sent: 50000  Rows_examined: 1350000
SET timestamp=1653555672;
SELECT hashId, hashlistId, hash, salt, plaintext, timeCracked, chunkId, isCracked, crackPos FROM Hash WHERE hashlistId='30' AND isCracked='0' ORDER BY Hash.hashId ASC LIMIT 1300000,50000;

# Query_time: 2.083822  Lock_time: 0.000003 Rows_sent: 50000  Rows_examined: 1400000
SET timestamp=1653555792;
SELECT hashId, hashlistId, hash, salt, plaintext, timeCracked, chunkId, isCracked, crackPos FROM Hash USE INDEX (hashlistId) WHERE hashlistId='30' AND isCracked='0' ORDER BY Hash.hashId ASC LIMIT 1350000,50000;

Could I find out if it is possible to modify the query to search by index of the table?

Hashtopolis: 0.12.0
Client: 0.6.0.10
Reply
#2
Hashtopolis in general is not meant normally to deal with so many multi-million Hashlists.

But your hint may be a good idea to add. Can you please open a feature request on Github, so we can put it on our to do list?
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)